8 Running test cases

Using package wyz.code.offensiveProgramming, you have the opportunity to define test cases and to embed them in your class definition, to ease retrieval and reuse.

Doing so, allows to get following benefits

1. discover defined test case definitions
2. run of any test case definition
3. get interactively the R code of a test case, allowing you to play with it, manually, when needed
4. get contextual results from the test case runs

8.1 Embedding test cases

Embedding test cases in class definition is accomplished by declaring a variable named testCaseDefinitions, and providing its content, that is a data.table. Content could be partial or complete depending of your goals. Spectrum of provided tests cases is as you desire it to be, as shallow or deep as needed.

The data.table must hold following columns and content

1. function_name, a vector of strings, each being the name of the function to test,
2. standard_evaluation, a vector of strings, where values are taken from set {‘correct’, ‘erroneous’, ‘failure’}
3. type_checking_enforcement, a vector of strings, where values are taken from set {‘correct’, ‘erroneous’, ‘failure’}
4. test case definitions, that is a list of TestCaseDefinition objects.

Correct implies right type and right result. Erroneous implies right type and wrong result. Failure implies wrong type. To get more details about syntax, please refer to manual page of TestCaseDefinition.

8.2 Discovering test cases

The considered R object has to be instrumented with test case definitions.

8.2.1 Test case verifications

To verify test cases definitions use function verifyTestCaseDefinitions.

verifyTestCaseDefinitions(AdditionTCFIG1())
#> $validity
#> [1] TRUE
#> 
#> $check
#> [1] "full instrumentation check"
#> 
#> $class
#> [1] "AdditionTCFIG1"
#> 
#> $intent
#> [1] "naming and instrumentation format and content seems good"
#> 
#> $message
#> [1] "success"

Here, the validity flag is TRUE and checks have been done against full instrumentation, meaning each function of the object must own at least one associated test case.

If this does not match your need, specify requiresFullInstrumentation_b_1 = FALSE as parameter to the call.

8.2.2 Getting instrumented functions

Use function retrieveTestCaseDefinitions to get access to recorded test case definitions

tc <- retrieveTestCaseDefinitions(AdditionTCFIG1())

if (is.data.table(tc)) {
  tc[, .N, by = 'function_name']
}
#>      function_name N
#> 1:       addDouble 9
#> 2:      addInteger 9
#> 3:    divideByZero 3
#> 4: generateWarning 1
#> 5:   generateError 1
#> 6:  addMultiDouble 3
#> 7: addMultiInteger 3

8.2.2.1 No test cases instrumentation

retrieveTestCaseDescriptions(Addition())
#> [1] "provided object owns no test case definitions"

8.2.3 Getting test cases descriptions

Use function retrieveTestCaseDescriptions to get access to recorded test case descriptions.

tc <- retrieveTestCaseDescriptions(AdditionTCFIG1())

if (is.data.table(tc)) {
  cat(nrow(tc), 'test cases\n')
  tc[, .N, by = 'function_name']
}
#> 29 test cases
#>      function_name N
#> 1:       addDouble 9
#> 2:      addInteger 9
#> 3:    divideByZero 3
#> 4: generateWarning 1
#> 5:   generateError 1
#> 6:  addMultiDouble 3
#> 7: addMultiInteger 3

Getting test case description for function divideByZero, can be achieved with a sequence like

tc[function_name == 'divideByZero']
#>    function_name description
#> 1:  divideByZero       1 / 0
#> 2:  divideByZero      -1 / 0
#> 3:  divideByZero       0 / 0

Of course, no obligation to restrict output to a single function as in this example.

Do not be fooled, description is text not code as you could deduce from previous example. See below.

tc[function_name == 'generateWarning']
#>      function_name      description
#> 1: generateWarning generate warning

8.3 Running test cases

8.3.1 Run a single test case

I want to run the test number 4 of object AdditionTCFIG1. Let’s first, know more about it.

test_to_run <- 4
obj <- AdditionTCFIG1() # instanciate your object
tc <- retrieveTestCaseDefinitions(obj)[test_to_run]
print(tc)
#>    function_name standard_evaluation type_checking_enforcement
#> 1:     addDouble             correct                   correct
#>               test_case
#> 1: <TestCaseDefinition>
tc[1]$test_case[[1]]
#> $params
#> $params[[1]]
#> [1] NaN
#> 
#> $params[[2]]
#> [1] NaN
#> 
#> 
#> $expected_result
#> [1] NaN
#> 
#> $description
#> [1] "sum 2 NAN"

To run the chosen test case, use function runTestCase.

rtc <- runTestCase(obj, test_to_run, emo$type)
rtc$synthesis
#>    status                      mode index value_check function_return_check
#> 1:   TRUE type_checking_enforcement     4        TRUE                  TRUE
#>    parameter_check expected_evaluation execution_evaluation failure_origin
#> 1:            TRUE             correct              correct           <NA>

Result has two parts. A raw part, that holds the intermediate computation results, and a synthesis part, that is the only one shown above.

Here, the status flag is TRUE meaning test case behaves perfectly in accordance with declared semantic names, under provided values.

8.3.2 Run several test cases

You can provide a vector instead of a single test number if you want to run several use test cases in one call.

test_to_run <- 12:14
tc <- retrieveTestCaseDefinitions(obj)[test_to_run]
print(tc)
#>    function_name standard_evaluation type_checking_enforcement
#> 1:    addInteger             correct                   correct
#> 2:    addInteger             correct                   correct
#> 3:    addInteger           erroneous                   failure
#>               test_case
#> 1: <TestCaseDefinition>
#> 2: <TestCaseDefinition>
#> 3: <TestCaseDefinition>
lapply(seq_len(nrow(tc)), function(k) print(tc[k]$test_case[[1]]))
#> $params
#> $params[[1]]
#> [1] NA
#> 
#> $params[[2]]
#> [1] NA
#> 
#> 
#> $expected_result
#> [1] NA
#> 
#> $description
#> [1] "sum 2 NA_integer"
#> 
#> $params
#> $params[[1]]
#> [1] 45
#> 
#> $params[[2]]
#> [1] 44
#> 
#> 
#> $expected_result
#> [1] 89
#> 
#> $description
#> [1] "sum a converted string with one integer"
#> 
#> $params
#> $params[[1]]
#> [1] 34
#> 
#> $params[[2]]
#> [1] 44.5
#> 
#> 
#> $expected_result
#> [1] 78
#> 
#> $description
#> [1] "sum 1 integer and 1 double"
#> [[1]]
#> [[1]]$params
#> [[1]]$params[[1]]
#> [1] NA
#> 
#> [[1]]$params[[2]]
#> [1] NA
#> 
#> 
#> [[1]]$expected_result
#> [1] NA
#> 
#> [[1]]$description
#> [1] "sum 2 NA_integer"
#> 
#> 
#> [[2]]
#> [[2]]$params
#> [[2]]$params[[1]]
#> [1] 45
#> 
#> [[2]]$params[[2]]
#> [1] 44
#> 
#> 
#> [[2]]$expected_result
#> [1] 89
#> 
#> [[2]]$description
#> [1] "sum a converted string with one integer"
#> 
#> 
#> [[3]]
#> [[3]]$params
#> [[3]]$params[[1]]
#> [1] 34
#> 
#> [[3]]$params[[2]]
#> [1] 44.5
#> 
#> 
#> [[3]]$expected_result
#> [1] 78
#> 
#> [[3]]$description
#> [1] "sum 1 integer and 1 double"

rtc <- runTestCase(obj, test_to_run, emo$type)
rtc$synthesis
#>    status                      mode index value_check function_return_check
#> 1:   TRUE type_checking_enforcement    12        TRUE                  TRUE
#> 2:   TRUE type_checking_enforcement    13        TRUE                  TRUE
#> 3:  FALSE type_checking_enforcement    14       FALSE                 FALSE
#>    parameter_check expected_evaluation execution_evaluation
#> 1:            TRUE             correct              correct
#> 2:            TRUE             correct              correct
#> 3:           FALSE             failure              failure
#>                             failure_origin
#> 1:                                    <NA>
#> 2:                                    <NA>
#> 3: value check, function return type check

Test 14 fails under chosen evaluation mode, and therefore requires a fix.

Here looking at raw results will provide root causes of failure. Provided values are double, whereas integers were expected. To understand the full details, you will have to run this example and to print rtc globally to get access to many details far too long to be displayed here.