stresstest_exercise {exams}

Stress Testing Exercises

Description

In order to check the correct behavior of an exercise it is compiled several times. In each iteration the objects created by the exercise are collected and its values can be inspected.

Usage

## Stresstest function.
stresstest_exercise(file, n = 100, verbose = TRUE, seeds = NULL,
  stop_on_error = length(as.character(unlist(file))) < 2, ...)

## Plotting stresstest results.
## S3 method for class 'stress'
plot(x, type = c("overview", "solution", "rank", "runtime"),
  threshold = NULL, variables = NULL,
  spar = TRUE, ask = TRUE, ...)

Arguments

file

character. A specification of an exercise file. If multiple files should be tested, argument file can also be a vector, matrix or list of files. The latter case sets argument plot = FALSE.

n

integer. The number of replications.

verbose

logical. Should the seeds used for compiling the exercise be prompted on the console.

seeds

The seeds that should be used when compiling the exercise. The default is seeds = 1:n.

stop_on_error

logical. Should the function stop on error or return the the seed, the file name and the error message. Useful when testing a number of exercises.

x

An object returned from function stresstest_exercise.

type

character. type == "overview" plots the basic overview, i.e, the runtimes, numeric solutions, position/number/rank of correct solution(s), if available. If type == "solution", the numeric solutions are plotted against all input parameters stored in the objects element of x. type == "rank" draws spineplots of ranks vs. input parameters stored in objects. type == "runtime" plots the compiling runtimes vs. objects.

threshold

numeric. Can be used to set a threshold, e.g., for numeric solutions a factor is created, factor(solution <= threshold), that is used on the y-axis of a spineplot.

variables

character. The variables that should be used from the objects for plotting.

spar

logical. Should graphical parameters be set or not.

ask

logical. For multiple plots, should the user be asked to hit the return key to see the next plot.

...

Arguments passed to xexams.

Details

In order to check the correct behavior of an exercise function stresstest_exercise runs xexams n times using different seeds. If an error occurs when compiling, the error can be reproduced by setting the seed that is prompted at the console and create the exercise again, e.g., with exams2html. This way errors can be detected systematically.

All objects with length 1, which are created by the exercise, are collected in a data frame. These objects are assumed to be input parameters that control the output of the exercise. This can be used to detect certain input values that, e.g., lead to very long run times, or drive the number of correct answers in multiple choice exercises, etc.

For single and multiple choice type question the position(s) of the correct solution(s) is returned. For single choice questions that are created from a numeric version, e.g., using function num_to_schoice the answers are again converted to numeric and the rank of the correct solution is reported. The rank is sometimes heavily driven by some input parameters, e.g., the correct solution is always the largest or the smallest. For non-numeric choice questions, the rank is based on the lexicographical order of the answerlist.

Value

Function stresstest_exercise returns an object of class "stress" (a named list) with the following elements:

seeds

The seeds that where used.

runtime

Compiling times for each iteration.

objects

A data frame of length 1 objects that are created by the exercise.

solution

The numeric solution, availability is depending on the type of exercise.

position

A matrix indicating the position of correct solutions.

rank

The rank of the correct solution, only available for choice exercises.

ntrue

The number of correct answers in multiple choice type questions.

See Also

xexams, num_to_schoice

Examples

## Not run: ## Stress testing.
t1 <- stresstest_exercise("tstat.Rmd", n = 100)
t2 <- stresstest_exercise("tstat2.Rmd", n = 100)

## Plotting.
plot(t1, type = "overview")
plot(t1, type = "solution")
plot(t1, type = "solution", threshold = 30)
plot(t2, type = "rank")
plot(t2, type = "runtime")

## For custom inspection, object can be
## transformed to a data.frame.
head(as.data.frame(t2))

## Multiple testing.
files <- list(
  "boxplots.Rmd",
  c("tstat.Rmd", "ttest.Rmd", "confint.Rmd"),
  c("regression.Rmd", "anova.Rmd"),
  "scatterplot.Rmd",
  "relfreq.Rmd"
)
t3 <- stresstest_exercise(files, n = 100)
plot(t3)

## End(Not run)