performanceResample {BuyseTest}

Uncertainty About Performance of a Classifier (EXPERIMENTAL)

Description

Use resampling to quantify uncertainties about the performance of one or several binary classifiers evaluated via cross-validation.

Usage

performanceResample(
  object,
  data = NULL,
  name.response = NULL,
  type.resampling = "permutation",
  n.resampling = 1000,
  fold.repetition = 0,
  conf.level = 0.95,
  cpus = 1,
  seed = NULL,
  trace = TRUE,
  filename = NULL,
  ...
)

Arguments

object

a glm or range object, or a list of such object.

data

[data.frame] the training data.

name.response

[character] The name of the response variable (i.e. the one containing the categories).

type.resampling

[character] Should non-parametric bootstrap ("bootstrap") or permutation of the outcome ("permutation") be used.

n.resampling

[integer,>0] Number of bootstrap samples or permutations.

fold.repetition

[integer,>0] Nnumber of folds used in the cross-validation. Should be strictly positive.

conf.level

[numeric, 0-1] confidence level for the confidence intervals.

cpus

[integer, >0] the number of CPU to use. If strictly greater than 1, resampling is perform in parallel.

seed

[integer, >0] Random number generator (RNG) state used when starting resampling. If NULL no state is set.

trace

[logical] Should the execution of the function be traced.

filename

[character] Prefix for the files containing each result.

...

arguments passed to performance.

Details

WARNING: using bootstrap after cross-validation may not provide valid variance/CI/p-value estimates.

Value

An S3 object of class performance.