9 Generating testthat test files

When your R code is offensive programming instrumented, it becomes possible to generate testthat unit test files, thus improving greatly developers productivity.

To be able to generate testthat unit test files, you must ensure that your R code is function return type instrumented and test cases instrumented. Both are required for this generation.

9.1 Package wyz.code.testthat in action

9.1.1 Setting up the context

To create testthat unit test files, you must provide a target folder, to store the generated unit test files. In this session, generated files will be stored onto folder generated-testthat.

You may change this, but be warned that generated files might be overwritten without any reminder.

Be careful, if you set the target folder to tests/testthat as you may loose previous work. You may save frequently your results in configuration management to be able to retrieve original file content whenever required.

library(wyz.code.testthat)

target_folder <- 'generated-testthat'
if (!dir.exists(target_folder)) dir.create(target_folder)

generateTests <- function(sourceFile_s_1, sourcePackage_s_1, object_o_1) {
  g <- gautfo(object_o_1, sourceFile_s_1, sourcePackage_s_1, target_folder)
  print(g)
  g
}

Function generateTests is just a wrapper which prints the computed results.

9.1.2 Loading objects

Here, I reuse 4 files from package wyz.code.offensiveProgramming to generate unit test files from.

source_package <- 'wyz.code.offensiveProgramming'

source_files <- c(
  'code-samples/both-defs/good/full/AdditionTCFIG1.R',
  'code-samples/no-defs/Addition.R',
  'code-samples/frt-defs/good/partial/AdditionFIPartial.R',
  'code-samples/tcd-defs/good/partial/AdditionTCPartial.R'
)

invisible(sapply(source_files, function(e) {
  source(system.file(e, package = source_package))
}))

Objects are chosen to cover some very common cases and to show variability of results when generating testthat test cases from.

object	information
AdditionTCFIG1	Fully offensive programming instrumented
Addition	no offensive programming instrumentation
AdditionFIPartial.R	partial function return type definition, no test case definitions
AdditionTCPartial.R	no function return type definition, partial test case definitions

9.1.3 Unit test file generation

Generation is done on a per class basis and is achieved using function gautfo or function generateAllUnitTestsFromObject.

9.1.3.1 Nominal case

Class AdditionTCFIG1 is used. Generation is quite straightforward.

print(gautfo(AdditionTCFIG1(), source_files[1], source_package, target_folder))
#> $class
#> [1] "AdditionTCFIG1"
#> 
#> $filenames
#>                                                    filename overwritten
#> 1:       generated-testthat/test_AdditionTCFIG1-addDouble.R        TRUE
#> 2:      generated-testthat/test_AdditionTCFIG1-addInteger.R        TRUE
#> 3:    generated-testthat/test_AdditionTCFIG1-divideByZero.R        TRUE
#> 4: generated-testthat/test_AdditionTCFIG1-generateWarning.R        TRUE
#> 5:   generated-testthat/test_AdditionTCFIG1-generateError.R        TRUE
#> 6:  generated-testthat/test_AdditionTCFIG1-addMultiDouble.R        TRUE
#> 7: generated-testthat/test_AdditionTCFIG1-addMultiInteger.R        TRUE

A file is created for each function for which test case definitions are defined. This file contains all the generated tests for two evaluation modes that are standard_R_evaluation and type_checking_enforcement. Each mode get its own testthat context.

Results is a list with two entries. Entry named filenames holds a data.table providing insight about testthat compliant created files.

9.1.3.2 Exception cases

All other objects do not reach miminum level of offensive programming instrumentation to allow test generation. Expected results is no testthat unit test file generation.

print(gautfo(Addition(), source_files[2], source_package, target_folder))
#> [1] "Class [Addition] apparently owns no test instrumentation. No test created."

print(gautfo(AdditionFIPartial(), source_files[3], source_package, target_folder))
#> [1] "Class [AdditionFIPartial] apparently owns no test instrumentation. No test created."

print(gautfo(AdditionTCPartial(), source_files[4], source_package, target_folder))
#> [1] "Class [AdditionTCPartial] apparently owns no function return type instrumentation. No test created."

9.2 Generated unit test file content

Typically, generated R code will looks like following unit test code. Note, that comments are provided to ease cross-referencing and to link back easily to wyz.code.offensiveProgramming test case number. This is helpful when facing dysfunctions.

source(system.file("code-samples/both-defs/good/full/AdditionTCFIG1.R",
                   package = "wyz.code.offensiveProgramming"))
object_o_1 <- AdditionTCFIG1()
emsre <- EvaluationMode("standard_R_evaluation")
rtcsre24 <- runTestCase(object_o_1, 24, emsre)
rtcsre25 <- runTestCase(object_o_1, 25, emsre)
rtcsre26 <- runTestCase(object_o_1, 26, emsre)

test_that('addMultiDouble', {

# test 24 - sum of 1 integer and 1 double - correct
expect_true(rtcsre24$synthesis$status)
expect_true(rtcsre24$synthesis$value_check)


# test 25 - sum of 1 double, 2 integers and 1 NA_integer_ - correct
expect_true(rtcsre25$synthesis$status)
expect_true(rtcsre25$synthesis$value_check)


# test 26 - sum of nothing - correct
expect_true(rtcsre26$synthesis$status)
expect_true(rtcsre26$synthesis$value_check)

})

emtce <- EvaluationMode("type_checking_enforcement")
rtctce24 <- runTestCase(object_o_1, 24, emtce)
rtctce25 <- runTestCase(object_o_1, 25, emtce)
rtctce26 <- runTestCase(object_o_1, 26, emtce)

test_that('addMultiDouble', {

# test 24 - sum of 1 integer and 1 double - correct
expect_true(rtctce24$synthesis$status)
expect_true(rtctce24$synthesis$value_check)


# test 25 - sum of 1 double, 2 integers and 1 NA_integer_ - correct
expect_true(rtctce25$synthesis$status)
expect_true(rtctce25$synthesis$value_check)


# test 26 - sum of nothing - correct
expect_true(rtctce26$synthesis$status)
expect_true(rtctce26$synthesis$value_check)

})

9.3 Caveats

Generation of unit test file uses meta-programmation based on call function, and aims to produce R valid code. Indeed, format and presentation are not managed, in generated file. You may use RStudio editing facilities to ensure nice presentation, although neither mandatory nor required.

Generated tests cases are ready to run. Use the standard way to run your testthat test cases onto them. If you face some test failures, verify following points.

1. ensure offensive programming evaluation of your code is running fine for ALL evaluation schemes, and that there exists no residual issues prior to generate test files
2. ensure your generated test files are well up to date with the your R offensive programming code. You may regenerate your tests files at any time if you have any doubt.

Note that unit test file generated is fully dependent of your R source and of the instrumented scope. If there are function not instrumented in your source code, do not expect to have unit test cases for them. If you plan to use (ref:tt), then go for fully instrumented offensive programming classes. This will help you achieve results faster. One severe issue of partial compliance, is that you have to remember all the exceptions. When dealing with few classes, quite easy, when dealing with several classes, it becomes messy to keep a clear view of what is instrumented or not. Not because of lack of tools, simply because you cannot anymore keep it present in your mind.

Also note, when working incrementally, i.e. creating some code, generating related testthat test files, and iterating the procedure, you need to regenerate the unit test cases each time you change the R source code or the offensive programming instrumentation at the scope of changes. If you change 2 classes, regenerate the test files of the 2 classes. Again, keeping the books of such approch is heavy and cumbersome. You would better go for systematic and global testthat test file generation, except if you modified manually produced results.

Best way to put wyz.code.testthat in practice, is to apply following procedure

1. apply offensive programming at the required scope and ensure wyz.code.offensiveProgramming test cases are valid, using runTestCase function
2. generate testthat test cases in one single pass using wyz.code.testthat. To do so, create an R script. This will ease your pain, and will provide consistent results through calls while allowing replay at any time,
3. apply testthat testing practice, to verify that generated tests are running fine.