pr.test {usefun}

Compare two Precision-Recall curves

Description

Test the hypothesis that the true difference in PR AUCs is equal to 0. We implement the same bootstrap method based on the idea from pROC::roc.test(). The PR AUC is calculated using PRROC::pr.curve() with the interpolation method of Davis (2006).

Usage

pr.test(
  labels,
  pred1,
  pred2,
  boot.n = 10000,
  boot.stratified = TRUE,
  alternative = "two.sided"
)

Arguments

labels

numeric()
Vector of responses/labels (only two classes/values allowed: cases/positive class = 1 and controls/negative class = 0)

pred1

numeric()
Vector of prediction values. Higher values denote positive class.

pred2

numeric()
Vector of prediction values. Higher values denote positive class. Must have the same length as pred1.

boot.n

numeric(1)
Number of bootstrap resamples. Default: 10000

boot.stratified

logical(1)
Whether the bootstrap resampling is stratified (same number of cases/controls in each replicate as in the original sample) or not. Advised to use especially when classes from labels are imbalanced. Default: TRUE.

alternative

character(1)
Specifies the alternative hypothesis. Either "two.sided", "less" or "greater". Default: "two.sided".

Value

a list with the AUCs of the two original prediction vectors and the p-value of the bootstrap-based test.

References

Davis J, Goadrich M (2006). “The relationship between precision-recall and ROC curves.” Proceedings of the 23rd International Conference on Machine Learning, 148(4), 233–240. doi:10.1145/1143844.1143874.

Examples

set.seed(42)
# imbalanced labels
labels = sample(c(0,1), 20, replace = TRUE, prob = c(0.8,0.2))
# predictions
pred1 = rnorm(20)
pred2 = rnorm(20)
pr.test(labels, pred1, pred2, boot.n = 1000, boot.stratified = FALSE)
pr.test(labels, pred1, pred2, boot.n = 1000, boot.stratified = TRUE)