pt_oob {dsos}

Permutation Test With Out-Of-Bag Scores

Description

Test for no adverse shift with outlier scores. Like goodness-of-fit testing, this two-sample comparison takes the training set, x_train as the as the reference. The method checks whether the test set, x_test, is worse off relative to this reference set. The function scorer assigns an outlier score to each instance/observation in both training and test set.

Usage

pt_oob(x_train, x_test, scorer, n_pt = 2000)

Arguments

x_train

Training (reference/validation) sample.

x_test

Test sample.

scorer

Function which returns a named list with outlier scores from the training and test sample. The first argument to scorer must be x_train; the second, x_test. The returned named list contains two elements: train and test, each of which is a vector of corresponding (outlier) scores. See notes below for more information.

n_pt

The number of permutations.

Details

The null distribution of the test statistic is based on n_pt permutations. For speed, this is implemented as a sequential Monte Carlo test with the simctest package. See Gandy (2009) for details. The prefix pt refers to permutation test. This approach does not use the asymptotic null distribution for the test statistic. This is the recommended approach for small samples. The test statistic is the weighted AUC (WAUC).

Value

A named list of class outlier.test containing:

• statistic: observed WAUC statistic

• seq_mct: sequential Monte Carlo test, when applicable

• p_value: p-value

• outlier_scores: outlier scores from training and test set

Notes

The scoring function, scorer, predicts out-of-bag scores to mimic out-of-sample behaviour. The suffix oob stands for out-of-bag to highlight this point. This out-of-bag variant avoids refitting the underlying algorithm from scorer at every permutation. It can, as a result, be computationally appealing.

References

Kamulete, V. M. (2022). Test for non-negligible adverse shifts. In The 38th Conference on Uncertainty in Artificial Intelligence. PMLR.

Gandy, A. (2009). Sequential implementation of Monte Carlo tests with uniformly bounded resampling risk. Journal of the American Statistical Association, 104(488), 1504-1511.

See Also

[pt_refit()] for (slower) p-value approximation via refitting. [at_oob()] for p-value approximation from asymptotic null distribution.

Other permutation-test: pt_from_os(), pt_refit()

Examples

library(dsos)
set.seed(12345)
data(iris)
idx <- sample(nrow(iris), 2 / 3 * nrow(iris))
iris_train <- iris[idx, ]
iris_test <- iris[-idx, ]
# Use a synthetic (fake) scoring function for illustration
scorer <- function(tr, te) list(train=runif(nrow(tr)), test=runif(nrow(te)))
pt_test <- pt_oob(iris_train, iris_test, scorer = scorer)
pt_test