testPackage {tinytest2JUnit}

Test an R package and report the results in JUnit

Description

Run all tests of a package and report the results as JUnit xml. This function can be seen as a drop in replacement for tinytest::test_package() but with a key difference that uncaught errors will be catched and reported JUnit! This function is intended to be used in a test stage of a CI build.

Usage

testPackage(
  pkgname,
  file = stdout(),
  errorOnFailure = TRUE,
  testdir = "tinytest",
  lib.loc = NULL,
  at_home = FALSE,
  ncpu = NULL,
  ...
)

Arguments

pkgname

character(1). Name of the package to tests.

file

character(1) | connection: Full file path or connection object to write the JUnit xml content to. By default stdout() connection is used. Warning if the file already exist it will be overwritten!

errorOnFailure

logical(1) Should an error be raised (after writing the JUnit) when a at least one test failed? By default TRUE. This is done as a convenience to have the CI fail at the test stage on failure.

testdir

character(1) testing directory of the package. See ?tinytest::test_package() for more details.

lib.loc

character(1) | NULL Library location where the package is installed. By default: NULL meaning the package is searched on the standard .libPaths().

at_home

logical(1) should local test be run? By default FALSE as we want to as closely mimic the environement of how tests would get ran in CRAN. See for more details tinytest::test_package().

ncpu

⁠postive integer(1) | clutser⁠ Either an integer specifying the amount of cpu's to parralize the testing over or a cluster object to run the tests in.

...

Extra arguments passed on to runTestDir()

Details

testPackage() is meant as a CI-friendly alternative to the native tinytest::test_package(). Next to directly reporting the tests results in a JUnit xml format, it also catches errors that are raised in the tests files and reports them as "error" in the JUnit.

tinytest::test_package() would have let the error bubble up, stop the testing process and not report any failures from other test files. One is then also forced to look into the logs of the CI to see what the error was. testPackage() presents you that error in the JUnit with a stacktrace. Next to all the test results of the other files that ran without a problem.

If you prefer the behaviour from tinytest::test_package(), you can still use it in combination with writeJUnit() if all tests results pass.

Just like tinytest::test_package() an error is raised if at least one failure occured during testing. Obviously catched errors are also seen as failures. This error is raised after the test results have been written away to the file, such that your CI can still pick it up and report the failure. The error raising is done as a convenience to stop the CI from continue if test-failure occured. You can opt-out of this behaviour by setting the errorOnFailure parameter to FALSE. Then a case tinytests2JUnit object is returned (a sub-class of tinytests object containing addition info for the JUnit). Caught errors are also captured in this object as tinytest-objects. They actually have a special sub-class but this is considered an internal implemenation detail.

testPackage() is NOT meant to be called from within your tests/tinytests.R file! Tests invoked by R CMD Check or on CRAN should still make use of tinytest::test_package(). This function is only meant to be called from within a testing step in your CI to report the test results in an JUnit xml format.

Value

If errorOnFailure = FALSE, a tinytests2JUnit object (a subclass of tinytests object that captures more info for export to JUnit). Else, an error is raised if at least on failure occurs. Meant as convenience to automatically stop the CI build.

Side-effects

Side effects are registered as 'passed' tests in the JUnit output and have been given a status "SIDE-EFFECT". The call and diff is also returned in the standard-output of the testcase tag.

They are not considred failures and would thus not stop a pipeline.

tinytests to JUnit

To comply the the JUnit specification the tests results are adapted as follows:

• A single test run tinytests is mapped to a ⁠<testsuites>⁠ tag.

• All tinytest results from a single file are mapped to a single ⁠<testsuite>⁠ tag.

  • The name of the testsuite is equal to the test file name (without the file suffix)

• An individual tinytest object (eg. a single ⁠except_*⁠ exception test) is mapped to a ⁠<testcase>⁠ tag.

  • The name of the testcase is equal to the fileName + Line specification of where the expect statement is performed + the info.

For reference: https://llg.cubic.org/docs/junit/

See Also

runTestDir() and tinytest::test_package().

Examples

tmpFile <- tempfile(fileext = ".xml")
testPackage("tinytest", file = tmpFile, verbose = 0)