10 Meta-testing
Meta-testing is the activity aiming to test a function while providing no data to test it.
In a R context, it means being able to
- discover function signature
- infer data type for each argument
- generate data set to be used for each argument
- run the function with generated data sets
- give back some summary statistics about discoveries of various test run achieved
If you try it by hand, you will probably succeed, because the second point will be managed directly by your brain. If you try it by a program, type inference is much trickier, because any argument in R could be of any type. Generally, you need the documentation and explanations to restrict the scope of possible types. That’s where using wyz.code.meta-testing will ease your work and bring instrumentation to get results in a more reliable and quicker way.
10.1 Traditional way of testing
By testing, I mean hand testing and operating scope and limits discovery.
10.1.1 Empirical approach
If you opt for an empirial testing of function cos, then you will enter command lines for each case you want to test. If you need many of them, it could become really boring very fast. It also disturbates you from the analysis of the results.
Moreover, in this approach you are nearly forced to open the black box cos and to get knowledge about it. That’s where it hurts, as working under predictable time in those conditions is really difficult.
10.1.2 More industrial approach
Going a little bit further than previous, you just use loops and scripts to ease replay. Let’s test base function cos in this way.
rt <- tryCatch( lapply(list(2 * pi / 1:9, NA, Inf, 1+1i, list(), letters[1:3]), cos),
error = function(e) { print(e); NaN } )
#> Warning in FUN(X[[i]], ...): NaNs produced
#> <simpleError in FUN(X[[i]], ...): non-numeric argument to mathematical function>And now, we have to unravel the arguments to find which ones are generating warnings or errors, to identify the ones that are accepted. Note that this trial is a gentle trial. It does not try for example to provide raw type or data frame or matrix or a function as argument.
Again, you will have to get knowledge about the function to be able to provide expected parameter types and values. You also would have to decipher the R documentation related to the function. Some documentations are easy and clear, some others much more difficult to understand and interpret.
10.2 Meta testing
10.2.1 Meta testing approach
Let’s test function op_cos using wyz.code.meta-testing.
The general procedure to follow is
- create a wrapper function if your function is not offensive programming instrumented
- get some knowlegde about the function signature complexity,
- fire massive tests, and get synthesis results
10.2.2 Create a wrapper function
Any R function can be classified as offensive programming compliant or not. Second case is indeed much more common and will be encountered more often.
In such a case, use offensiveProgrammingWrapFunction or function opwf to generate a new R function that will be offensive programming compliant.
You have to provide semantic argument names to this function to be able to generate correctly the wrapping function. Once done, type inference is now driven by semantic argument names.
Let’s see an example, considering function cos from base package.
library(wyz.code.metaTesting)
op_cos <- opwf(cos, 'radianAngleOrComplex_')
op_cos
#> function (radianAngleOrComplex_)
#> {
#> cos(radianAngleOrComplex_)
#> }
#> <environment: 0x557d990aaca0>You may wonder what are the difference between the standard R signature and this one? They share same number of arguments, just the name changed. Yes, but as the name is now a semantic name, it can be managed by a factory See FunctionParameterTypeFactory in wyz.code.offensiveProgramming for more details to generate data to match the parameters.
10.2.3 Getting some complexity knowledge
This is achieved using function computeArgumentsCombination.
Here there exist only one signature for function op_cos and so for function cos.
Note that function computeArgumentsCombination can be used with any R function. It does not requires offensive programming instrumentated function as argument.
10.2.4 Firing massive tests
This is achieved using function exploreSignatures.
The second argument passed to the function is just a type restriction to be enforced when generating data for testing purposes. Here I asked for integer real and complex mathematical types these are different from R integer, double and complex as they cannot take value NA. Type restrictions are only considered for polymorphic arguments reminder: the ones that ends with an underscore.
es <- exploreSignatures(op_cos, list(radianAngleOrComplex_ = c('im', 'r', 'cm')))
nm <- names(es$success$synthesis)
sx <- nm[startsWith(nm, 'number_suc')][1] # typo error number_sucessful corrected in v1.1.12 of metaTesting
cat('succesful tests:', es$success$synthesis[[sx]], '\n')
#> succesful tests: 6
cat('erroneous tests:', es$failure$synthesis$number_erroneous_tests, '\n')
#> erroneous tests: 6Results tell you that same signature brings various results. Here, 6 tests succeeded and 6 failed.
10.2.4.1 Result analysis
cat('error message:', es$failure$synthesis$error, '\n')
#> error message: Error in cos(radianAngleOrComplex_): non-numeric argument to mathematical function
#> All execution errors provided the same error message.
First diagnostic result, errors are tied to usage of non numeric values.
cat('succesful pattern:', es$success$synthesis$imperative, '\n')
#> succesful pattern: {homo,hetero}_{vector}_{one,two,three}
cat('erroneous pattern:', es$failure$synthesis$imperative, '\n')
#> erroneous pattern: {homo,hetero}_{list}_{one,two,three}It is clear that the issue is tied to using a list as value to an imperative argument. Looking closer, on success, you see that only vectors provided results.
Secon diagnostic results, list should not be used as input parameter.
Now, you can conclude that cos function
- accepts as input vectors of integers, reals and complex
- passing a list as argument brings an error with the shown message.
To be complete, note that
- as I enforced mathematical arguments, values NA, NaN and Inf are no more possible values for test. This match the mathematic function cosinus and not the R function cos. This is an important point, know what scope you want to test, not just what function you want to test
- I was expecting the cosinus of a complex number to compute the cosinus of the argument of the complex number, normalized by its modulus. Was expecting a ℂ to ℝ function. That is clearly not the case as output are complex numbers. That is a ℂ to ℂ function. Explanation of the behavior is that the function cosine is extended to complex numbers as explained here. If you read R documentation of cos function, you would not get any relevant information about this.
10.3 A more complex example
Let’s now use function append from base package.
10.3.1 Create a wrapper function
As you know, we need first to create the offensive programming wrapper function.
op_append <- opwf(append, c('originalValues_', 'valuesToInsert_', 'afterIndex_ui_1'))
op_append
#> function (originalValues_, valuesToInsert_, afterIndex_ui_1 = length(originalValues_))
#> {
#> append(originalValues_, valuesToInsert_, after = afterIndex_ui_1)
#> }
#> <environment: 0x557d97f161f8>As you can see, parameter substitution is also achieved in code for default arguments.
10.3.2 Getting some complexity knowledge
How complex is it to test this function?
rv <- computeArgumentsCombination(op_append)
rv$theoritical_signature_number
#> [1] 2
rv$signatures
#> [[1]]
#> [1] "originalValues_, valuesToInsert_"
#>
#> [[2]]
#> [1] "originalValues_, valuesToInsert_, afterIndex_ui_1"There are two call signatures, one without default parameter, one with. Let’s test them.
10.3.3 Firing massive tests
es <- exploreSignatures(op_append)
print(es$success$synthesis)
#> $number_successfull_tests
#> [1] 24
#>
#> $signatures
#> [1] "originalValues_, valuesToInsert_"
#> [2] "originalValues_, valuesToInsert_, afterIndex_ui_1"
#>
#> $imperative
#> [1] "{homo,hetero}_{vector,list}_{one,two,three}"
#>
#> $default
#> [1] "{none,full}"
print(es$failure$synthesis)
#> [1] NA10.3.3.1 Result analysis
From the 24 test runs, no errors where generated.
10.3.3.1.1 Instrospecting the results
If you are curious about a particular test call, let’s say number 22, just introspect returned values as below. You will see the code use to call the function during the test.
print(es$success$code[22]$call_string)
#> [1] "op_append(originalValues_ = list(c(-12.9267629287206, 6.28514515282586, ---9.57221709983423, 5.77918772771955), c(0+0i, -9+15i, 16+13i, ---1-2i)), valuesToInsert_ = list(list(data = list(c(5.72568594152108, ----0.64659017464146, 0.983572392724454), c(0+9i, -2-3i, -1-1i, ----15+14i, 9-8i, 5-15i), 3L)), structure(c(22198, 12808, 12808), class = \"Date\")), --- afterIndex_ui_1 = 7L)"If you desire to introspect the call results, use this approach.
print(es$success$code[22]$result)
#> [[1]]
#> [[1]][[1]]
#> [1] -12.926763 6.285145 9.572217 5.779188
#>
#> [[1]][[2]]
#> [1] 0+ 0i -9+15i 16+13i 1- 2i
#>
#> [[1]][[3]]
#> [[1]][[3]]$data
#> [[1]][[3]]$data[[1]]
#> [1] 5.7256859 -0.6465902 0.9835724
#>
#> [[1]][[3]]$data[[2]]
#> [1] 0+ 9i -2- 3i -1- 1i -15+14i 9- 8i 5-15i
#>
#> [[1]][[3]]$data[[3]]
#> [1] 3
#>
#>
#>
#> [[1]][[4]]
#> [1] "2030-10-11" "2005-01-25" "2005-01-25"10.4 An example using ellipsis
Let’s now use function sum from base package.
10.4.1 Create a wrapper function
As you know, we need first to create the offensive programming wrapper function.
op_sum <- opwf(sum, c('...', 'removeNAValues_b_1'))
op_sum
#> function (..., removeNAValues_b_1 = FALSE)
#> {
#> sum(..., na.rm = removeNAValues_b_1)
#> }
#> <environment: 0x557d9a545238>As you can see, parameter substitution is also achieved in code for default arguments.
10.4.2 Getting some complexity knowledge
How complex is it to test this function?
cac_sum <- computeArgumentsCombination(op_sum)
print(cac_sum$signatures)
#> [[1]]
#> character(0)
#>
#> [[2]]
#> [1] "removeNAValues_b_1"
#>
#> [[3]]
#> [1] "ellipsis1_"
#>
#> [[4]]
#> [1] "ellipsis1_, removeNAValues_b_1"
#>
#> [[5]]
#> [1] "ellipsis1_, ellipsis2_"
#>
#> [[6]]
#> [1] "ellipsis1_, ellipsis2_, removeNAValues_b_1"
#>
#> [[7]]
#> [1] "ellipsis1_, ellipsis2_, ellipsis3_"
#>
#> [[8]]
#> [1] "ellipsis1_, ellipsis2_, ellipsis3_, removeNAValues_b_1"There are eight call signatures, four without default parameter, four with. By default, ellipsis is replaced by 0 to three arguments. That’s why, first signature is empty, and the total is 8. Let’s test them.
10.4.3 Firing massive tests
es <- exploreSignatures(op_sum)
print(es$success$synthesis)
#> $number_successfull_tests
#> [1] 8
#>
#> $signatures
#> [1] "no argument signature" "removeNAValues_b_1"
#>
#> $ellipsis
#> [1] "{homo,hetero}_{vector,list}_{none}"
#>
#> $default
#> [1] "{none,full}"
print(es$failure$synthesis)
#> $number_erroneous_tests
#> [1] 24
#>
#> $error
#> [1] "Error in sum(..., na.rm = removeNAValues_b_1): invalid 'type' (list) of argument\n"
#>
#> $signatures
#> [1] "ellipsis1_"
#> [2] "ellipsis1_, removeNAValues_b_1"
#> [3] "ellipsis1_, ellipsis2_"
#> [4] "ellipsis1_, ellipsis2_, removeNAValues_b_1"
#> [5] "ellipsis1_, ellipsis2_, ellipsis3_"
#> [6] "ellipsis1_, ellipsis2_, ellipsis3_, removeNAValues_b_1"
#>
#> $ellipsis
#> [1] "{homo,hetero}_{vector,list}_{one,two,three}"
#>
#> $default
#> [1] "{none,full}"From the 32 test runs, 8 passed, 24 failed. As we gave no restriction types for ellipsis, it has been replaced by any kind of types, and in particular some that cannot fit a sum. Let’s restrict the types to uses and run again same kind of test.
Much better. Still 32 tests, now 20 passed, 12 failed. Why?
10.4.3.1 Result analysis
print(es2$failure$synthesis)
#> $number_erroneous_tests
#> [1] 12
#>
#> $error
#> [1] "Error in sum(..., na.rm = removeNAValues_b_1): invalid 'type' (list) of argument\n"
#>
#> $signatures
#> [1] "ellipsis1_"
#> [2] "ellipsis1_, removeNAValues_b_1"
#> [3] "ellipsis1_, ellipsis2_"
#> [4] "ellipsis1_, ellipsis2_, removeNAValues_b_1"
#> [5] "ellipsis1_, ellipsis2_, ellipsis3_"
#> [6] "ellipsis1_, ellipsis2_, ellipsis3_, removeNAValues_b_1"
#>
#> $ellipsis
#> [1] "{homo,hetero}_{list}_{one,two,three}"
#>
#> $default
#> [1] "{none,full}"All failures seems to be related to arguments passed as list.
10.5 A tricky example
Let’s now use function kronecker from base package.
10.5.1 Create wrapper function
As you know, we need first to create the offensive programming wrapper function.
op_kronecker <- opwf(kronecker, c('arrayA_a_1', 'arrayB_a_1', 'function_f_1',
'computeDimensionNames_b_1', '...'))
op_kronecker
#> function (arrayA_a_1, arrayB_a_1, function_f_1 = "*", computeDimensionNames_b_1 = FALSE,
#> ...)
#> {
#> kronecker(arrayA_a_1, arrayB_a_1, FUN = function_f_1, make.dimnames = computeDimensionNames_b_1,
#> ...)
#> }
#> <environment: 0x557d9acaef18>10.5.2 Getting some complexity knowledge
How complex is it to test this function?
cac_kronecker <- computeArgumentsCombination(op_kronecker)
print(cac_kronecker$signatures)
#> [[1]]
#> [1] "arrayA_a_1, arrayB_a_1"
#>
#> [[2]]
#> [1] "arrayA_a_1, arrayB_a_1, ellipsis1_"
#>
#> [[3]]
#> [1] "arrayA_a_1, arrayB_a_1, ellipsis1_, ellipsis2_"
#>
#> [[4]]
#> [1] "arrayA_a_1, arrayB_a_1, ellipsis1_, ellipsis2_, ellipsis3_"
#>
#> [[5]]
#> [1] "arrayA_a_1, arrayB_a_1, computeDimensionNames_b_1"
#>
#> [[6]]
#> [1] "arrayA_a_1, arrayB_a_1, computeDimensionNames_b_1, ellipsis1_"
#>
#> [[7]]
#> [1] "arrayA_a_1, arrayB_a_1, computeDimensionNames_b_1, ellipsis1_, ellipsis2_"
#>
#> [[8]]
#> [1] "arrayA_a_1, arrayB_a_1, computeDimensionNames_b_1, ellipsis1_, ellipsis2_, ellipsis3_"
#>
#> [[9]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1"
#>
#> [[10]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1, ellipsis1_"
#>
#> [[11]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1, ellipsis1_, ellipsis2_"
#>
#> [[12]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1, ellipsis1_, ellipsis2_, ellipsis3_"
#>
#> [[13]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1"
#>
#> [[14]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1, ellipsis1_"
#>
#> [[15]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1, ellipsis1_, ellipsis2_"
#>
#> [[16]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1, ellipsis1_, ellipsis2_, ellipsis3_"There are sixteen call signatures for which meta testing must generate data.
10.5.3 Firing massive tests
Let’s try brut force analysis first.
tryCatch(es <- exploreSignatures(op_kronecker),
error = function(e) print(e) )
#> <simpleError in abort("no draw function associated with", strBracket(value_s_1[1])): no draw function associated with [a]>This fails as there exist no data generation function provided for array. We have to provide one by recording it into the data factory. To do so, here is the procedure to follow
- create the draw function that will generate the data type you need and that is not yet recorded into the data factory
- test it unitary and ensure result is of good type
- record new types into the data factory
- make your factory findable
- fire tests
# step 1 - create the draw function - wrong way
wrong_draw_integer_array <- function(n, replace_b_1 = TRUE) {
m <- n + sample(0:3, 1)
matrix(seq(1, n * m), byrow = TRUE, nrow = n,
dimnames = list(paste('row_', 1:n), paste('col_', 1:m)))
}
# wrong because it does not respect argument names that must be
# n_i_1 and replace_b_1
# step 2 - test draw function
a1 <- wrong_draw_integer_array(2)
# step 3 - record new type into data factory - failure here
df <- retrieveDataFactory()
df$addSuffix('a', 'array', wrong_draw_integer_array)
#> [1] FALSE
# step 1 - create the draw factory - right way
draw_integer_array <- function(n_i_1, replace_b_1 = TRUE) {
m <- n_i_1 + sample(0:3, 1)
matrix(seq(1, n_i_1 * m), byrow = TRUE, nrow = n_i_1,
dimnames = list(paste('row_', 1:n_i_1), paste('col_', 1:m)))
}
# step 1 - create draw function
draw_function <- function(n_i_1, replace_b_1 = TRUE) { list(`*`, `+`, `-`)[[sample(1:3, 1)]]}
# step - 2 test good practice verifies your functions behave correctly on a single example
a1 <- draw_integer_array(2)
a2 <- draw_integer_array(3)
f <- draw_function(1)
kronecker(a1, a2, f, TRUE)
#> col_ 1:col_ 1 col_ 1:col_ 2 col_ 1:col_ 3 col_ 1:col_ 4
#> row_ 1:row_ 1 1 2 3 4
#> row_ 1:row_ 2 6 7 8 9
#> row_ 1:row_ 3 11 12 13 14
#> row_ 2:row_ 1 6 12 18 24
#> row_ 2:row_ 2 36 42 48 54
#> row_ 2:row_ 3 66 72 78 84
#> col_ 1:col_ 5 col_ 2:col_ 1 col_ 2:col_ 2 col_ 2:col_ 3
#> row_ 1:row_ 1 5 2 4 6
#> row_ 1:row_ 2 10 12 14 16
#> row_ 1:row_ 3 15 22 24 26
#> row_ 2:row_ 1 30 7 14 21
#> row_ 2:row_ 2 60 42 49 56
#> row_ 2:row_ 3 90 77 84 91
#> col_ 2:col_ 4 col_ 2:col_ 5 col_ 3:col_ 1 col_ 3:col_ 2
#> row_ 1:row_ 1 8 10 3 6
#> row_ 1:row_ 2 18 20 18 21
#> row_ 1:row_ 3 28 30 33 36
#> row_ 2:row_ 1 28 35 8 16
#> row_ 2:row_ 2 63 70 48 56
#> row_ 2:row_ 3 98 105 88 96
#> col_ 3:col_ 3 col_ 3:col_ 4 col_ 3:col_ 5 col_ 4:col_ 1
#> row_ 1:row_ 1 9 12 15 4
#> row_ 1:row_ 2 24 27 30 24
#> row_ 1:row_ 3 39 42 45 44
#> row_ 2:row_ 1 24 32 40 9
#> row_ 2:row_ 2 64 72 80 54
#> row_ 2:row_ 3 104 112 120 99
#> col_ 4:col_ 2 col_ 4:col_ 3 col_ 4:col_ 4 col_ 4:col_ 5
#> row_ 1:row_ 1 8 12 16 20
#> row_ 1:row_ 2 28 32 36 40
#> row_ 1:row_ 3 48 52 56 60
#> row_ 2:row_ 1 18 27 36 45
#> row_ 2:row_ 2 63 72 81 90
#> row_ 2:row_ 3 108 117 126 135
#> col_ 5:col_ 1 col_ 5:col_ 2 col_ 5:col_ 3 col_ 5:col_ 4
#> row_ 1:row_ 1 5 10 15 20
#> row_ 1:row_ 2 30 35 40 45
#> row_ 1:row_ 3 55 60 65 70
#> row_ 2:row_ 1 10 20 30 40
#> row_ 2:row_ 2 60 70 80 90
#> row_ 2:row_ 3 110 120 130 140
#> col_ 5:col_ 5
#> row_ 1:row_ 1 25
#> row_ 1:row_ 2 50
#> row_ 1:row_ 3 75
#> row_ 2:row_ 1 50
#> row_ 2:row_ 2 100
#> row_ 2:row_ 3 150
# step 3 - record new data types into data factory - success here
df$addSuffix('a', 'array', draw_integer_array)
#> [1] TRUE
df$addSuffix('f', 'function', draw_function)
#> [1] TRUE
# step 4 - make your factory findable
options(op_mt_data_factory = df)
# step 5 - fire tests - up to 768 contexts managed in one shot
es <- exploreSignatures(op_kronecker)10.5.3.1 Result analysis
Computed results helps greatly analysis.
nm <- names(es$success$synthesis)
sx <- nm[startsWith(nm, 'number_suc')][1] # typo error number_sucessful corrected in v1.1.12 of metaTesting
cat('succesful tests:', es$success$synthesis[[sx]], '\n')
#> succesful tests: 144
cat('erroneous tests:', es$failure$synthesis$number_erroneous_tests, '\n')
#> erroneous tests: 432
print(es$failure$synthesis$error)
#> [1] "Error in outer(X, Y, FUN, ...): using ... with FUN = \"*\" is an error\n"
#> [2] "Error in FUN(X, Y, ...): operator needs one or two arguments\n"Here number of successful test is already very significant, although overcame by number of failure tests.
cat('succesful pattern:', es$success$synthesis$imperative, '\n')
#> succesful pattern: {homo,hetero}_{vector,list}_{one,two,three}
cat('erroneous pattern:', es$failure$synthesis$imperative, '\n')
#> erroneous pattern: {homo,hetero}_{vector,list}_{one,two,three}Patterns for success and failure are the same. So, issue should logically come from data types.
cat('succesful signatures\n')
#> succesful signatures
print(es$success$synthesis$signatures)
#> [1] "arrayA_a_1, arrayB_a_1"
#> [2] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1"
cat('erroneous signatures\n')
#> erroneous signatures
print(es$failure$synthesis$signatures)
#> [1] "arrayA_a_1, arrayB_a_1, ellipsis1_"
#> [2] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1, ellipsis1_"
#> [3] "arrayA_a_1, arrayB_a_1, ellipsis1_, ellipsis2_"
#> [4] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1, ellipsis1_, ellipsis2_"
#> [5] "arrayA_a_1, arrayB_a_1, ellipsis1_, ellipsis2_, ellipsis3_"
#> [6] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1, ellipsis1_, ellipsis2_, ellipsis3_"Analysis of signatures brings more insight as all the error signatures implies ellipsis. Such result is great as it provides very quickly a good hint about the root cause.
10.6 Pitfalls to avoid
Some common and well-known pitfalls are
- When using opwf function, make sure you provide the argument names in the right order. Examine created function signature prior going further. Make sure it fits the desired definition you look for.
- DataFactory changes remain invisible to processing as long as you do not set the option op_mt_data_factory with the name of the R variable that holds the DataFactory you want to use. This is often forgotten.