pairwise.test {IOHanalyzer}

Performs a pairwise Kolmogorov-Smirnov test on the bootstrapped running times among a data set

Description

This function performs a Kolmogorov-Smirnov test on each pair of algorithms in the input x to determine which algorithm gives a significantly smaller running time. The resulting p-values are arranged in a matrix, where each cell (i, j) contains a p-value from the test with alternative hypothesis: the running time of algorithm i is smaller (thus better) than that of j.

Usage

pairwise.test(x, ...)

## S3 method for class 'list'
pairwise.test(x, max_eval, bootstrap.size = 30, ...)

## S3 method for class 'DataSetList'
pairwise.test(x, ftarget, bootstrap.size = 0, which = "by_FV", ...)

Arguments

x

either a list that contains running time sample for each algorithm as sub-lists, or a DataSetList object

...

all other options

max_eval

list that contains the maximal running time for each algorithm as sub-lists

bootstrap.size

integer, the size of the bootstrapped sample. Set to 0 to disable bootstrapping

ftarget

float, the target value used to determine the running / hitting

which

wheter to do fixed-target ('by_FV') or fixed-budget ('by_RT') comparison time

Value

A matrix containing p-values of the test

Examples

pairwise.test(subset(dsl, funcId == 1), 16)