benchmark {benchr}

High Precise Measurement of R Expressions Execution Time

Description

benchmark serves as a more accurate replacement of the often seen system.time(replicate(1000, expr)) expression. It tries hard to accurately measure only the time it takes to evaluate expr. To achieve this, the sub-millisecond (supposedly nanosecond) accurate timing functions most modern operating systems provide are used. Additionally all evaluations of the expressions are done in C++ code to minimize any measurment error.

Usage

benchmark(
  ...,
  times = 100L,
  order = c("random", "inorder", "block"),
  envir = parent.frame(),
  progress = TRUE,
  gcFirst = TRUE,
  gcDuring = FALSE
)

Arguments

...

Captures any number of unevaluated expressions passed to benchmark as named or unnamed arguments.

times

Integer. Number of times to evaluate each expression.

order

Character. The order in which the expressions are evaluated.

envir

The environment in which the expressions will be evaluated.

progress

Logical. Show progress bar during expressions evaluation.

gcFirst

Logical. Should a garbage collection be performed immediately before the timing?

gcDuring

Logical. Should a garbage collection be performed immediately before each iteration, as produced by times? (very slow)

Details

Before evaluating each expression times times, the overhead of calling the timing functions and the C++ function call overhead are estimated. This estimated overhead is subtracted from each measured evaluation time. Should the resulting timing be negative, a warning is thrown and the respective value is replaced by 0. If the timing is zero, a warning is also raised. Should all evaluations result in one of the two error conditions described above, an error is raised.

Value

Object of class benchmark, which is a data.frame with a number of additional attributes and contains the following columns:

expr

The deparsed expression as passed to benchmark or the name of the argument if the expression was passed as a named argument.

time

The measured execution time of the expression in seconds. The order of the observations in the data frame is the order in which they were executed.

An object of class benchmark also contains the following attributes:

precision

Timer precision in seconds.

error

Timer error (overhead) in seconds.

units

Units for time intervals (by default, "s" – seconds).

times

Number of repeats for each measurement.

order

Execution regime.

gc

Whether garbage collection took place before each execution.

The order in which the expressions are evaluated

“random”

(the default) randomizes the execution order

“inorder”

executes each expression in order

“block”

executes all repetitions of each expression as one block.

Author(s)

Artem Klevtsov [e-mail]

See Also

summary.benchmark(), mean.benchmark(), print.benchmark(), plot.benchmark(), boxplot.benchmark()

Examples

## Measure the time it takes to dispatch a simple function call
## compared to simply evaluating the constant NULL
f <- function() NULL
res <- benchmark(NULL, f(), times = 1000L)

## Print results:
print(res)

## Plot results
boxplot(res)