bf_compare {dsos}  R Documentation 
Bayesian and Frequentist Test from Outlier Scores
Description
Test for no adverse shift with outlier scores. Like goodnessoffit testing,
this twosample comparison takes the training (outlier) scores,
os_train
, as the reference. The method checks whether the test
scores, os_test
, are worse off relative to the training set.
Usage
bf_compare(os_train, os_test, threshold = 1/12, n_pt = 4000)
Arguments
os_train 
Outlier scores in training (reference) set. 
os_test 
Outlier scores in test set. 
threshold 
Threshold for adverse shift. Defaults to 1 / 12, the asymptotic value of the test statistic when the two samples are drawn from the same distribution. 
n_pt 
The number of permutations. 
Details
This compares the Bayesian to the frequentist approach for convenience.
The Bayesian test mimics 'bf_from_os()' and the frequentist one,
'pt_from_os()'. The Bayesian test computes Bayes factors based on the
asymptotic (defaults to 1/12) and the exchangeable threshold. The latter
calculates the threshold as the median weighted AUC (WAUC) after n_pt
permutations assuming outlier scores are exchangeable. This is recommended
for small samples. The frequentist test converts the onesided (onetailed)
pvalue to the Bayes factor  see as_bf
function.
Value
A list of factors (BF) for 3 different test specifications:

frequentist
: Frequentist BF. 
bayes_noperm
: Bayestion BF test with asymptotic threshold. 
bayes_perm
: Bayestion BF with exchangeable threshold.
Notes
The outlier scores should all mimic outofsample behaviour. Mind that the training scores are not insample and thus, biased (overfitted) while the test scores are outofsample. The mismatch – insample versus outofsample scores – voids the test validity. A simple fix for this is to get the training scores from an indepedent (fresh) validation set; this follows the train/validation/test sample splitting convention and the validation set is effectively the reference set or distribution in this case.
See Also
[bf_from_os()] for bayes factor, the Bayesian test. [pt_from_os()] for pvalue, the frequentist test.
Other bayesiantest:
as_bf()
,
as_pvalue()
,
bf_from_os()
Examples
library(dsos)
set.seed(12345)
os_train < rnorm(n = 100)
os_test < rnorm(n = 100)
bayes_test < bf_compare(os_train, os_test)
bayes_test
# To run in parallel on local cluster, uncomment the next two lines.
# library(future)
# future::plan(future::multisession)
parallel_test < bf_compare(os_train, os_test)
parallel_test