kfold_ttest {correctR}

Compute correlated t-statistic and p-value for k-fold cross-validated results

Description

Compute correlated t-statistic and p-value for k-fold cross-validated results

Usage

kfold_ttest(x, y, n, k, tailed = c("two", "one"), greater = NULL)

Arguments

x

numeric vector of values for model A

y

numeric vector of values for model B

n

integer denoting total sample size

k

integer denoting number of folds used in k-fold

tailed

character denoting whether to perform a two-tailed or one-tailed test. Can be one of "two" or "one". Defaults to "two"

greater

character specifying whether "x" or "y" is greater for the one-tailed test if tailed = "one". Defaults to NULL

Value

data.frame containing the test statistic and p-value

Author(s)

Trent Henderson

References

Nadeau, C., and Bengio, Y. Inference for the Generalization Error. Machine Learning 52, (2003).

Corani, G., Benavoli, A., Demsar, J., Mangili, F., and Zaffalon, M. Statistical comparison of classifiers through Bayesian hierarchical modelling. Machine Learning, 106, (2017).

Examples

x <- rnorm(100, mean = 95, sd = 0.5)
y <- rnorm(100, mean = 90, sd = 1)
kfold_ttest(x = x, y = y, n = 100, k = 5, tailed = "two")