at_from_os {dsos}  R Documentation 
Asymptotic Test from Outlier Scores
Description
Test for no adverse shift with outlier scores. Like goodnessoffit testing,
this twosample comparison takes the training set, x_train
as the
as the reference. The method checks whether the test set, x_test
, is
worse off relative to this reference set. The function scorer
assigns
an outlier score to each instance/observation in both training and test set.
Usage
at_from_os(os_train, os_test)
Arguments
os_train 
Outlier scores in training (reference) set. 
os_test 
Outlier scores in test set. 
Details
Li and Fine (2010) derives the asymptotic null distribution for the weighted
AUC (WAUC), the test statistic. This approach does not use permutations
and can, as a result, be much faster because it sidesteps the need to refit
the scoring function scorer
. This works well for large samples. The
prefix at stands for asymptotic test to tell it apart from the
prefix pt, the permutation test.
Value
A named list of class outlier.test
containing:

statistic
: observed WAUC statistic 
seq_mct
: sequential Monte Carlo test, when applicable 
p_value
: pvalue 
outlier_scores
: outlier scores from training and test set
Notes
The outlier scores should all mimic outofsample behaviour. Mind that the training scores are not insample and thus, biased (overfitted) while the test scores are outofsample. The mismatch – insample versus outofsample scores – voids the test validity. A simple fix for this is to get the training scores from an indepedent (fresh) validation set; this follows the train/validation/test sample splitting convention and the validation set is effectively the reference set or distribution in this case.
References
Kamulete, V. M. (2022). Test for nonnegligible adverse shifts. In The 38th Conference on Uncertainty in Artificial Intelligence. PMLR.
Gandy, A. (2009). Sequential implementation of Monte Carlo tests with uniformly bounded resampling risk. Journal of the American Statistical Association, 104(488), 15041511.
See Also
[at_oob()] for variant requiring a scoring function. [pt_from_os()] for permutation test with the outlier scores.
Other asymptotictest:
at_oob()
Examples
library(dsos)
set.seed(12345)
os_train < rnorm(n = 100)
os_test < rnorm(n = 100)
test_result < at_from_os(os_train, os_test)
test_result