10 Meta-testing

Meta-testing is the activity aiming to test a function while providing no data to test it.

In a R context, it means being able to

  1. discover function signature
  2. infer data type for each argument
  3. generate data set to be used for each argument
  4. run the function with generated data sets
  5. give back some summary statistics about discoveries of various test run achieved

If you try it by hand, you will probably succeed, because the second point will be managed directly by your brain. If you try it by a program, type inference is much trickier, because any argument in R could be of any type. Generally, you need the documentation and explanations to restrict the scope of possible types. That’s where using wyz.code.meta-testing will ease your work and bring instrumentation to get results in a more reliable and quicker way.

10.1 Wrapper function creation

Any R function can be classified as offensive programming compliant or not. Second case is indeed much more common and will be encountered more often.

In such a case, use offensiveProgrammingWrapFunction or function opwf to generate a new R function that will be offensive programming compliant.

You have to provide semantic argument names to this function to be able to generate correctly the wrapping function. Once done, type inference is now driven by semantic argument names.

Let’s see a trivial example, considering function cos from base package.

You may wonder what are the difference between the two signatures? They share same number of arguments, just the name changed. Yes, but as the name is now a semantic name, it can be managed by a factory See FunctionParameterTypeFactory in wyz.code.offensiveProgramming for more details.

10.2 Exploration function

Let’s test base function cos in a traditional way.

And now, we have to unravel the arguments to find which ones are generating warnings or errors, to identify the ones that are accepted. Note that this trial is a gentle trial. It does not try for example to provide raw type or data frame or matrix or a function as argument. If you opt for a more traditionnal approach, then you will have to type as many lines as value types you want to test. This could become really boring very fast, as it disturbates you from the analysis of the results.

Let’s test it using wyz.code.meta-testing.

First thing do achieve is to get some knowlegde about the function signature complexity. This is easily achieved using function computeArgumentsCombination.

The number of signatures is given by the signatures name. Here there exist only one signature for function op_cos and so for function cos.

Note that function computeArgumentsCombination can be used with any R function. It does not requires offensive programming instrumentated function as argument.

Let’s test function op_cos.

The second argument is just type restriction to be enforced. Here I asked for integer real and complex mathematical types these are different from R integer, double and complex as they cannot take value NA. Type restrictions are only considered for polymorphic arguments reminder: the ones that ends with an underscore.

Results tell you that same signature brings various results. Here, 6 tests succeeded and 6 failed. It is clear that the issue is tied to using a list as value to an imperative argument you will have to learn how to interpret the synthesis indeed. All execution errors provided the same error message. Looking closer, on success, you see that only vectors provided results.

Now, you can conclude that cos functions

  1. accepts as input vectors of integers, reals and complex
  2. passing a list as argument brings an error with the shown message.

To be complete, note that

  1. as I enforced mathematical arguments, values NA, NaN and Inf are no more possible values for test. This match the mathematic function cosinus and not the R function cos. This is an important point, know what scope you want to test, not just what function you want to test
  2. I do not understand the results when using complex numbers. I was expecting the cosinus of a complex number to compute the cosinus of the argument of the complex number, normalized by its modulus. Was expecting a ℂ to ℝ function. That is clearly not the case as output are complex numbers. That is a ℂ to ℂ function. I am still a taker of any explanation about this.

10.3 A more complex example

Let’s now use function append from base package.

As you know, we need first to create the offensive programming wrapper function.

As you can see, parameter substitution is also achieved in code for default arguments.

How complex is it to test this function?

There are two call signatures, one without default parameter, one with. Let’s test them.

From the 24 test runs, no errors where generated. If you are curious about a particular test call, let’s say number 22, just introspect returned values as below. You will see the code use to call the function during the test.

print(es$success$code[22]$call_string)
#> [1] "op_append(originalValues_ = list(c(-8+3i, 5+6i, -13+3i, 8+1i), ---    c(-8-6i, 15+5i, 5+8i, 12-5i)), valuesToInsert_ = list(c(17L, ---11L), list(data = list(list(data = list(as.raw(0x6e), list(data = list(---    list(data = list(c(8+12i, -2-8i), structure(16384, class = \"Date\"), ---        list(data = list(4+15i, list(data = list(list(data = list(---            6L, list(data = list(list(data = list(list(data = list(---                c(FALSE, NA, FALSE, NA), list(data = list(-5:-4, ---                  c(FALSE, NA, NA))), structure(c(972864000, ---                972864000), class = c(\"POSIXct\", \"POSIXt\")), ---                c(NA, 1.00412152381614, 1.00412152381614))))), ---                list(data = list(c(17L, 14L, 7L, 5L, 2L, 11L), ---                  list(data = list(list(data = list(2.69039925839752, ---                    c(-3+9i, -9-16i, 2-15i), c(13.2084962166846, ---                    13.3436268009245, 2.62213358655572, 6.90973385795951, ---                    7.19652696698904), c(10L, -17L))))), c(8.62264013290405, ---                  10.4135163128376, 8.36220926046371, 1.94969380274415, ---                  2.62334731966257, 8.64920395612717))))), TRUE, ---            list(data = list(list(data = list(TRUE, 7.38372031599283, ---                c(-13L, -4L, -9L))), c(-12L, -3L, -2L), list(---                data = list(NA)))), c(7.27329816808924, 3.28533956455067, ---            -11.548649426084, -5.39954257849604, -0.158040850423276---            ), c(6L, 9L, 4L))))), list(data = list(8.27031458774582, ---            list(data = list(list(data = list(integer(0))))))))), ---        c(\"oxmblc\", \"ppyed\", \"zwiiktp\", \"ppmzgz\", \"roaodslbtts\", ---        \"avxw\"), -15.2460190393031, list(data = list(list(data = list(---            numeric(0), integer(0))), list(data = list(c(\"wst\", ---        \"mvfbyevtob\", \"lmkhwocpf\"), as.raw(c(0x78, 0x64, 0x68, ---        0x69, 0x77, 0x68)), c(11.9735112781636, -8.66431576246396, ---        -15.858928845264), 2L, list(data = list(8L, list(data = list(---            c(\"viszsesun\", \"flgv\", \"iqpzh\", \"isziiphi\", \"aco\", ---            \"vqrufyq\"), c(NA, 2.33606173284352, -16.2792672854848, ---            -16.2792672854848, -2.4392126603052), numeric(0), ---            list(data = list(-11.1560563221574)), TRUE, -2.62471943721175)))), ---            list(data = list(list(data = list(c(-9.52451307699084, ---            -11.9914163388312))), FALSE)))), list(data = list(---            c(-4L, 11L, -1L, -12L, 3L), c(-16L, 17L, -16L, -1L, ---            7L), \"mfkmdh\", numeric(0), list(data = list(list(---                data = list(c(4.92849926324561, 13.471843149513, ---                -0.60454114805907), c(10.7167961113155, 7.43848054483533---                ))), c(\"rxoxh\", \"sihls\"), complex(0), c(6.67615007981658, ---            8.0937023460865, 13.4478220082819, 13.1799479089677---            ))), list(data = list(c(-4L, 3L), TRUE, list(data = list(---                structure(1466035200, class = c(\"POSIXct\", \"POSIXt\"---                )), NA)), list(data = list(list(data = list(c(NA, ---            \"eljucvwdth\"), FALSE, complex(0), c(11L, 2L, 16L, ---            2L, -15L))), c(-9.51196666806936, -4.35927340388298, ---            -7.8687717653811), list(data = list(list(data = list(---                c(4L, 6L, 3L, 7L, 4L), c(TRUE, TRUE), c(1.069873646833, ---                -8.54819805454463, -11.9553949269466, -14.8211299027316, ---                -8.02534481231123))), c(14L, -5L, 4L, -15L), ---                structure(20694, class = \"Date\"), c(4.46488302946091, ---                -14.9756830153055, -14.9756830153055, -14.9756830153055---                ), c(3.96209045872092, 5.82415194809437, 2.05197109282017---                ))), list(data = list(c(TRUE, FALSE, FALSE, FALSE, ---            TRUE), integer(0))))), list(data = list(c(17L, 10L, ---            6L, 13L))))), FALSE)))), c(\"ifkho\", \"ltts\", \"dazpi\", ---        \"ojmli\", \"uuomdhcrm\"))))))), list(data = list(c(3L, 12L, ---11L))), c(-7+10i, 10-10i, -10+6i, 8-2i), list(data = list(c(-5.65178386494517, ----13.7241868562996, -15.842213910073), FALSE, TRUE, c(-7L, -5L, ----9L, -9L)))))), afterIndex_ui_1 = 17L)"

If you desire to introspect the call results, use this approach.

print(es$success$code[22]$result)
#> [[1]]
#> [[1]][[1]]
#> [1]  -8+3i   5+6i -13+3i   8+1i
#> 
#> [[1]][[2]]
#> [1] -8-6i 15+5i  5+8i 12-5i
#> 
#> [[1]][[3]]
#> [1] 17 11
#> 
#> [[1]][[4]]
#> [[1]][[4]]$data
#> [[1]][[4]]$data[[1]]
#> [[1]][[4]]$data[[1]]$data
#> [[1]][[4]]$data[[1]]$data[[1]]
#> [1] 6e
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[1]]
#> [1]  8+12i -2- 8i
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[2]]
#> [1] "2014-11-10"
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[1]]
#> [1] 4+15i
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[1]]
#> [1] 6
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[1]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[1]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[1]]$data[[1]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[1]]$data[[1]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[1]]$data[[1]]$data[[1]]
#> [1] FALSE    NA FALSE    NA
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[1]]$data[[1]]$data[[2]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[1]]$data[[1]]$data[[2]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[1]]$data[[1]]$data[[2]]$data[[1]]
#> [1] -5 -4
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[1]]$data[[1]]$data[[2]]$data[[2]]
#> [1] FALSE    NA    NA
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[1]]$data[[1]]$data[[3]]
#> [1] "2000-10-30 01:00:00 CET" "2000-10-30 01:00:00 CET"
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[1]]$data[[1]]$data[[4]]
#> [1]       NA 1.004122 1.004122
#> 
#> 
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[2]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[2]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[2]]$data[[1]]
#> [1] 17 14  7  5  2 11
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[2]]$data[[2]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[2]]$data[[2]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[2]]$data[[2]]$data[[1]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[2]]$data[[2]]$data[[1]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[2]]$data[[2]]$data[[1]]$data[[1]]
#> [1] 2.690399
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[2]]$data[[2]]$data[[1]]$data[[2]]
#> [1] -3+ 9i -9-16i  2-15i
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[2]]$data[[2]]$data[[1]]$data[[3]]
#> [1] 13.208496 13.343627  2.622134  6.909734  7.196527
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[2]]$data[[2]]$data[[1]]$data[[4]]
#> [1]  10 -17
#> 
#> 
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[2]]$data[[2]]$data[[3]]
#> [1]  8.622640 10.413516  8.362209  1.949694  2.623347  8.649204
#> 
#> 
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[3]]
#> [1] TRUE
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[4]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[4]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[4]]$data[[1]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[4]]$data[[1]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[4]]$data[[1]]$data[[1]]
#> [1] TRUE
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[4]]$data[[1]]$data[[2]]
#> [1] 7.38372
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[4]]$data[[1]]$data[[3]]
#> [1] -13  -4  -9
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[4]]$data[[2]]
#> [1] -12  -3  -2
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[4]]$data[[3]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[4]]$data[[3]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[4]]$data[[3]]$data[[1]]
#> [1] NA
#> 
#> 
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[5]]
#> [1]   7.2732982   3.2853396 -11.5486494  -5.3995426  -0.1580409
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[2]]$data[[1]]$data[[6]]
#> [1] 6 9 4
#> 
#> 
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[3]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[3]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[3]]$data[[1]]
#> [1] 8.270315
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[3]]$data[[2]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[3]]$data[[2]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[3]]$data[[2]]$data[[1]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[3]]$data[[2]]$data[[1]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[3]]$data[[3]]$data[[2]]$data[[1]]$data[[1]]
#> integer(0)
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[4]]
#> [1] "oxmblc"      "ppyed"       "zwiiktp"     "ppmzgz"      "roaodslbtts"
#> [6] "avxw"       
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[5]]
#> [1] -15.24602
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[1]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[1]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[1]]$data[[1]]
#> numeric(0)
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[1]]$data[[2]]
#> integer(0)
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[1]]
#> [1] "wst"        "mvfbyevtob" "lmkhwocpf" 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[2]]
#> [1] 78 64 68 69 77 68
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[3]]
#> [1]  11.973511  -8.664316 -15.858929
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[4]]
#> [1] 2
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[5]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[5]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[5]]$data[[1]]
#> [1] 8
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[5]]$data[[2]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[5]]$data[[2]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[5]]$data[[2]]$data[[1]]
#> [1] "viszsesun" "flgv"      "iqpzh"     "isziiphi"  "aco"       "vqrufyq"  
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[5]]$data[[2]]$data[[2]]
#> [1]         NA   2.336062 -16.279267 -16.279267  -2.439213
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[5]]$data[[2]]$data[[3]]
#> numeric(0)
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[5]]$data[[2]]$data[[4]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[5]]$data[[2]]$data[[4]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[5]]$data[[2]]$data[[4]]$data[[1]]
#> [1] -11.15606
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[5]]$data[[2]]$data[[5]]
#> [1] TRUE
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[5]]$data[[2]]$data[[6]]
#> [1] -2.624719
#> 
#> 
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[6]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[6]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[6]]$data[[1]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[6]]$data[[1]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[6]]$data[[1]]$data[[1]]
#> [1]  -9.524513 -11.991416
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[2]]$data[[6]]$data[[2]]
#> [1] FALSE
#> 
#> 
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[1]]
#> [1]  -4  11  -1 -12   3
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[2]]
#> [1] -16  17 -16  -1   7
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[3]]
#> [1] "mfkmdh"
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[4]]
#> numeric(0)
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[5]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[5]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[5]]$data[[1]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[5]]$data[[1]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[5]]$data[[1]]$data[[1]]
#> [1]  4.9284993 13.4718431 -0.6045411
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[5]]$data[[1]]$data[[2]]
#> [1] 10.716796  7.438481
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[5]]$data[[2]]
#> [1] "rxoxh" "sihls"
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[5]]$data[[3]]
#> complex(0)
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[5]]$data[[4]]
#> [1]  6.676150  8.093702 13.447822 13.179948
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[1]]
#> [1] -4  3
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[2]]
#> [1] TRUE
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[3]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[3]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[3]]$data[[1]]
#> [1] "2016-06-16 02:00:00 CEST"
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[3]]$data[[2]]
#> [1] NA
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[1]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[1]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[1]]$data[[1]]
#> [1] NA           "eljucvwdth"
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[1]]$data[[2]]
#> [1] FALSE
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[1]]$data[[3]]
#> complex(0)
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[1]]$data[[4]]
#> [1]  11   2  16   2 -15
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[2]]
#> [1] -9.511967 -4.359273 -7.868772
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[3]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[3]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[3]]$data[[1]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[3]]$data[[1]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[3]]$data[[1]]$data[[1]]
#> [1] 4 6 3 7 4
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[3]]$data[[1]]$data[[2]]
#> [1] TRUE TRUE
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[3]]$data[[1]]$data[[3]]
#> [1]   1.069874  -8.548198 -11.955395 -14.821130  -8.025345
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[3]]$data[[2]]
#> [1]  14  -5   4 -15
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[3]]$data[[3]]
#> [1] "2026-08-29"
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[3]]$data[[4]]
#> [1]   4.464883 -14.975683 -14.975683 -14.975683
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[3]]$data[[5]]
#> [1] 3.962090 5.824152 2.051971
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[4]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[4]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[4]]$data[[1]]
#> [1]  TRUE FALSE FALSE FALSE  TRUE
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[4]]$data[[4]]$data[[2]]
#> integer(0)
#> 
#> 
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[5]]
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[5]]$data
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[6]]$data[[5]]$data[[1]]
#> [1] 17 10  6 13
#> 
#> 
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[6]]$data[[3]]$data[[7]]
#> [1] FALSE
#> 
#> 
#> 
#> 
#> 
#> [[1]][[4]]$data[[1]]$data[[2]]$data[[1]]$data[[7]]
#> [1] "ifkho"     "ltts"      "dazpi"     "ojmli"     "uuomdhcrm"
#> 
#> 
#> 
#> 
#> 
#> 
#> 
#> [[1]][[4]]$data[[2]]
#> [[1]][[4]]$data[[2]]$data
#> [[1]][[4]]$data[[2]]$data[[1]]
#> [1]  3 12 11
#> 
#> 
#> 
#> [[1]][[4]]$data[[3]]
#> [1]  -7+10i  10-10i -10+ 6i   8- 2i
#> 
#> [[1]][[4]]$data[[4]]
#> [[1]][[4]]$data[[4]]$data
#> [[1]][[4]]$data[[4]]$data[[1]]
#> [1]  -5.651784 -13.724187 -15.842214
#> 
#> [[1]][[4]]$data[[4]]$data[[2]]
#> [1] FALSE
#> 
#> [[1]][[4]]$data[[4]]$data[[3]]
#> [1] TRUE
#> 
#> [[1]][[4]]$data[[4]]$data[[4]]
#> [1] -7 -5 -9 -9

10.4 An example using ellipsis

Let’s now use function sum from base package.

As you know, we need first to create the offensive programming wrapper function.

As you can see, parameter substitution is also achieved in code for default arguments.

How complex is it to test this function?

There are eight call signatures, four without default parameter, four with. By default, ellipsis is replaced by 0 to three arguments. That’s why, first signature is empty, and the total is 8. Let’s test them.

From the 32 test runs, 8 passed, 24 failed. As we gave no restriction types for ellipsis, it has been replaced by any kind of characters, and in particular some that cannot fit a sum. Let’s restrict the types to uses and run again same kind of test.

Much better. Still 32 tests, now 20 passed, 12 failed. Why?

All failures seems to be related to arguments passed as list.

10.5 An example with imperative, default and ellipsis arguments

Let’s now use function kronecker from base package.

As you know, we need first to create the offensive programming wrapper function.

How complex is it to test this function?

cac_kronecker <- computeArgumentsCombination(op_kronecker)
print(cac_kronecker)
#> $names
#> $names$argument
#> [1] "arrayA_a_1" "arrayB_a_1"
#> 
#> $names$ellipsis
#> $names$ellipsis[[1]]
#> character(0)
#> 
#> $names$ellipsis[[2]]
#> [1] "ellipsis1_"
#> 
#> $names$ellipsis[[3]]
#> [1] "ellipsis1_, ellipsis2_"
#> 
#> $names$ellipsis[[4]]
#> [1] "ellipsis1_, ellipsis2_, ellipsis3_"
#> 
#> 
#> $names$default
#> $names$default[[1]]
#> character(0)
#> 
#> $names$default[[2]]
#> [1] "function_f_1"
#> 
#> $names$default[[3]]
#> [1] "computeDimensionNames_b_1"
#> 
#> $names$default[[4]]
#> [1] "function_f_1, computeDimensionNames_b_1"
#> 
#> 
#> 
#> $number
#> $number$argument
#> [1] 2
#> 
#> $number$ellipsis
#> [1] 0 1 2 3
#> 
#> $number$default
#> [1] 0 1 2
#> 
#> 
#> $signatures
#> $signatures[[1]]
#> [1] "arrayA_a_1, arrayB_a_1"
#> 
#> $signatures[[2]]
#> [1] "arrayA_a_1, arrayB_a_1, ellipsis1_"
#> 
#> $signatures[[3]]
#> [1] "arrayA_a_1, arrayB_a_1, ellipsis1_, ellipsis2_"
#> 
#> $signatures[[4]]
#> [1] "arrayA_a_1, arrayB_a_1, ellipsis1_, ellipsis2_, ellipsis3_"
#> 
#> $signatures[[5]]
#> [1] "arrayA_a_1, arrayB_a_1, computeDimensionNames_b_1"
#> 
#> $signatures[[6]]
#> [1] "arrayA_a_1, arrayB_a_1, computeDimensionNames_b_1, ellipsis1_"
#> 
#> $signatures[[7]]
#> [1] "arrayA_a_1, arrayB_a_1, computeDimensionNames_b_1, ellipsis1_, ellipsis2_"
#> 
#> $signatures[[8]]
#> [1] "arrayA_a_1, arrayB_a_1, computeDimensionNames_b_1, ellipsis1_, ellipsis2_, ellipsis3_"
#> 
#> $signatures[[9]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1"
#> 
#> $signatures[[10]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1, ellipsis1_"
#> 
#> $signatures[[11]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1, ellipsis1_, ellipsis2_"
#> 
#> $signatures[[12]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1, ellipsis1_, ellipsis2_, ellipsis3_"
#> 
#> $signatures[[13]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1"
#> 
#> $signatures[[14]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1, ellipsis1_"
#> 
#> $signatures[[15]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1, ellipsis1_, ellipsis2_"
#> 
#> $signatures[[16]]
#> [1] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1, ellipsis1_, ellipsis2_, ellipsis3_"
#> 
#> 
#> $theoritical_signature_number
#> [1] 16

There are sixteen call signatures.

Let’s try brut force analysis first.

This fails as there exist no data generation function provided for array. We have to provide one.

wrong_draw_integer_array <- function(n, replace_b_1 = TRUE) {
  m <- n + sample(0:3, 1)
  matrix(seq(1, n * m), byrow = TRUE, nrow = n, 
         dimnames = list(paste('row_', 1:n), paste('col_', 1:m)))
} 
# wrong because it does not respect argument names that must be
# n_i_1 and replace_b_1

df <- DataFactory()
df$addSuffix('a', 'array', wrong_draw_integer_array)
#> [1] FALSE

draw_integer_array <- function(n_i_1, replace_b_1 = TRUE) {
  m <- n_i_1 + sample(0:3, 1)
  matrix(seq(1, n_i_1 * m), byrow = TRUE, nrow = n_i_1, 
         dimnames = list(paste('row_', 1:n_i_1), paste('col_', 1:m)))
}

draw_function <- function(n_i_1, replace_b_1 = TRUE) { list(`*`, `+`, `-`)[[sample(1:3, 1)]]}

# good practice verifies your functions behave correctly on a single example
a1 <- draw_integer_array(2)
a2 <- draw_integer_array(3)
f <- draw_function(1)
kronecker(a1, a2, f, TRUE)
#>               col_ 1:col_ 1 col_ 1:col_ 2 col_ 1:col_ 3 col_ 1:col_ 4
#> row_ 1:row_ 1             2             3             4             5
#> row_ 1:row_ 2             6             7             8             9
#> row_ 1:row_ 3            10            11            12            13
#> row_ 2:row_ 1             5             6             7             8
#> row_ 2:row_ 2             9            10            11            12
#> row_ 2:row_ 3            13            14            15            16
#>               col_ 2:col_ 1 col_ 2:col_ 2 col_ 2:col_ 3 col_ 2:col_ 4
#> row_ 1:row_ 1             3             4             5             6
#> row_ 1:row_ 2             7             8             9            10
#> row_ 1:row_ 3            11            12            13            14
#> row_ 2:row_ 1             6             7             8             9
#> row_ 2:row_ 2            10            11            12            13
#> row_ 2:row_ 3            14            15            16            17
#>               col_ 3:col_ 1 col_ 3:col_ 2 col_ 3:col_ 3 col_ 3:col_ 4
#> row_ 1:row_ 1             4             5             6             7
#> row_ 1:row_ 2             8             9            10            11
#> row_ 1:row_ 3            12            13            14            15
#> row_ 2:row_ 1             7             8             9            10
#> row_ 2:row_ 2            11            12            13            14
#> row_ 2:row_ 3            15            16            17            18

# register functions
df$addSuffix('a', 'array', draw_integer_array)
#> [1] TRUE
df$addSuffix('f', 'function', draw_function)
#> [1] TRUE

# make your factory findable
Sys.setenv("OP_DATA_FACTORY" = "df")

# fire tests - up to 768 contexts managed in one shot
es <- exploreSignatures(op_kronecker)

print(es$success$synthesis)
#> $number_sucessfull_tests
#> [1] 144
#> 
#> $signatures
#> [1] "arrayA_a_1, arrayB_a_1"                                         
#> [2] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1"
#> 
#> $imperative
#> [1] "{homo,hetero}_{vector,list}_{one,two,three}"
#> 
#> $ellipsis
#> [1] "{homo,hetero}_{vector,list}_{none}"
#> 
#> $default
#> [1] "{none,partial,full}"
print(es$failure$synthesis)
#> $number_erroneous_tests
#> [1] 432
#> 
#> $error
#> [1] "Error in outer(X, Y, FUN, ...): using ... with FUN = \"*\" is an error\n" 
#> [2] "Error in FUN(X, Y, ...): l'opérateur a besoin d'un ou de deux arguments\n"
#> 
#> $signatures
#> [1] "arrayA_a_1, arrayB_a_1, ellipsis1_"                                                                 
#> [2] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1, ellipsis1_"                        
#> [3] "arrayA_a_1, arrayB_a_1, ellipsis1_, ellipsis2_"                                                     
#> [4] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1, ellipsis1_, ellipsis2_"            
#> [5] "arrayA_a_1, arrayB_a_1, ellipsis1_, ellipsis2_, ellipsis3_"                                         
#> [6] "arrayA_a_1, arrayB_a_1, function_f_1, computeDimensionNames_b_1, ellipsis1_, ellipsis2_, ellipsis3_"
#> 
#> $imperative
#> [1] "{homo,hetero}_{vector,list}_{one,two,three}"
#> 
#> $ellipsis
#> [1] "{homo,hetero}_{vector,list}_{one,two,three}"
#> 
#> $default
#> [1] "{none,partial,full}"

10.6 Pitfalls to avoid

  1. When using opwf function, make sure you provide the argument names in the right order. Examine created function signature prior going further. Make sure it fits the desired definition you look for.
  2. DataFactory changes remain invisible to processing as long as you do not set the environment variable OP_DATA_FACTORY with the name of the R variable that holds the DataFactory you want to use. This is often forgotten.