R: Testing the statistical significance of predictors in random...

rfvimptest {rfvimptest}

R Documentation

Testing the statistical significance of predictors in random forests using sequential permutation testing

Description

Implements several strategies for testing the statistical significance of predictors in random forests using sequential permutation testing procedures based on the permutation variable importance measure. See Hapfelmeier et al. (2022) for details.

Usage

rfvimptest(
  data,
  yname,
  Mmax = 500,
  varnames = NULL,
  p0 = 0.06,
  p1 = 0.04,
  alpha = 0.05,
  beta = 0.2,
  A = 0.1,
  B = 10,
  h = 8,
  nperm = 1,
  ntree = 500,
  progressbar = TRUE,
  test = c("general", "twosample")[1],
  type = c("SPRT", "SAPT", "pval", "certain", "complete")[1],
  condinf = FALSE,
  ...
)

Arguments

`data`	A `data.frame` containing the variables in the model.
`yname`	Name of outcome variable.
`Mmax`	Maximum number of permutations used in each permutation test. Default is 500.
`varnames`	Optional. Names of the variables for which testing should be performed. By default all variables in `data` with the exception of the outcome variable are used.
`p0`	The value of the p-value in the null hypothesis (H0: p = p0) of SPRT and SAPT. Default is 0.06.
`p1`	The value of the p-value in the alternative hypothesis (H1: p = p1) of SPRT and SAPT. Default is 0.04.
`alpha`	The significance level of SPRT when p = p0. Also known as type I error. Default is 0.05.
`beta`	One minus the power of SPRT when p = p1. Also known as type II error. Default is 0.2.
`A`	The quantity A in the formula of SAPT. Default is 0.1 for a type I error of 0.05. Usually not changed by the user.
`B`	The quantity B in the formula of SAPT. Default is 10 (1/A) for a type I error of 0.05. Usually not changed by the user.
`h`	The quantity h in the formula for the sequential Monte Carlo p-value. The default value for h is 8. Larger values lead to more precise p-value estimates, but are computationally more expensive.
`nperm`	The numbers of permutations of the out-of-bag observations over which the results are averaged, when calculating the variable importance measure values. Default is 1. Larger values than 1 can only be considered when `condinf=TRUE`, that is, when using random forests with conditional inference trees (Hothorn et al., 2006) as base learners.
`ntree`	Number of trees per forest. Default is 500.
`progressbar`	Output the current progress of the calculations for each variable to the console? Default is TRUE.
`test`	Type of the permutation test to perform. This can be either "general" or "twosample", where "general" refers to the usual (sequential) permutation test and "twosample" refers to the two-sample (sequential) permutation test. For the latter, see also Coleman et al. (2019). Note, however, that "twosample" is experimental and should not be used for formal testing. See the details section below.
`type`	Type of the sequential method to use in the permutation tests. The choices are: "SPRT", "SAPT", "pval", "certain", and "complete". See the 'Details' section below for details.
`condinf`	Set this value to `TRUE` if random forests using conditional inference trees (Hothorn et al., 2006) should be used and to `FALSE` if classical random forests using CART trees should be used. Default is `FALSE`.
`...`	Further arguments passed to `ranger::ranger` (if `condinf=FALSE`) or `party::cforest_unbiased()` (if `condinf=TRUE`).

Details

Only the general permutation test (test="general") controls the type I error. In contrast, the two-sample permutation test (test="twosample") is associated with inflated type I error, which can lead to false positive findings. An advantage of the two-sample permutation test is that it is very fast. Therefore, this experimental approach may be used as an informal screening tool for finding informative variables. It is, however, not a valid testing procedure. Note also that the paper of Coleman et al. (2019) on which the two-sample test is based has not yet been published in a peer-reviewed journal and that the theory underlying this procedure might thus still need further review.

SRPT (type="SRPT") and SAPT (type="SAPT") are similar sequential procedures, where SRPT is faster with respect to accepting H0, that is, detecting non-informative variables, whereas SAPT is faster with respect to accepting H1, that is, detecting informative variables. Therefore, SRPT may be preferred for datasets with only few informative variables, whereas SAPT is preferable for datasets with many informative variables. The Monte Carlo p-value based testing procedure (type="pval") should be used, when p-values are required. The choice type="complete" offers a conventional permutation test (that is, without sequential testing) (Hapfelmeier and Ulm, 2013). This choice is computationally the most intensive. Lastly, the choice type="certain" is similar to type="complete", but performs early stopping by ending the permutation iterations as soon as it is certain which outcome the conventional permutation test would take. That is, type="certain" can be considered as a computationally more effective version of type="complete".

Value

Object of class rfvimptest with elements

`testtype`	Type of the permutation test performed and sequential method used.
`varimp`	Variable importance for each considered independent variable.
`testres`	The results ("keep H0" vs. "accept H1") of the tests for each considered independent variable.
`pvalues`	The p-values of the tests for each considered independent variable. Note that p-values are only obtained for the method types "pval" and "complete".
`stoppedearly`	For each independent variable, whether the calculations stopped early ("yes") or the maximum of `Mmax` permutations was reached ("no").
`perms`	The number of permutations performed for each independent variable.
`Mmax`	Maximum number of permutations used in each permutation test.
`ntree`	Number of trees per forest.
`comptime`	The time the computations needed.

Author(s)

Alexander Hapfelmeier, Roman Hornung

References

Breiman, L. (2001). Random forests. Mach Learn, 45:5-32, <doi: 10.1023/A:1010933404324>.
Coleman, T., Peng, W., Mentch, L. (2019). Scalable and efficient hypothesis testing with random forests. arXiv preprint arXiv:1904.07830, <doi: 10.48550/arXiv.1904.07830>.
Hapfelmeier, A., Hornung, R., Haller, B. (2022). Sequential Permutation Testing of Random Forest Variable Importance Measures. arXiv preprint arXiv:2206.01284, <doi: 10.48550/arXiv.2206.01284>.
Hapfelmeier, A., Ulm, K. (2013). A new variable selection approach using Random Forests. CSDA 60:50–69, <doi: 10.1016/j.csda.2012.09.020>.
Hapfelmeier, A., Hothorn, T., Ulm, K., Strobl, C. (2014). A new variable importance measure for random forests with missing data. Stat Comput 24:21–34, <doi: 10.1007/s11222-012-9349-1>.
Hothorn, T., Hornik, K., Zeileis, A. (2006). Unbiased Recursive Partitioning: A Conditional Inference Framework. J Comput Graph Stat 15(3):651–674, <doi: 10.1198/106186006X133933>.
Wright, M. N., Ziegler, A. (2017). ranger: A fast implementation of random forests for high dimensional data in C++ and R. J Stat Softw 77:1-17, <doi: 10.18637/jss.v077.i01>.

Examples



## Load package:
library("rfvimptest")

## Set seed to obtain reproducible results:
set.seed(1234)

# Load example data:
data(hearth2)

# NOTE: For illustration purposes a very small number (Mmax=20) of maximum
# permutations is considered. This number would be much too small for actual
# applications. The default number is Max=500.

# By default, SPRT is performed:
(ptest_sprt <- rfvimptest(data=hearth2, yname="Class", Mmax=20))
ptest_sprt$varimp
ptest_sprt$testres

# Calculation of p-values using the Monte Carlo p-value based testing procedure:
(ptest_pval <- rfvimptest(data=hearth2, yname="Class", type="pval", Mmax=20))
ptest_pval$pvalues

# If the frequency of informative variables is expected to be high SAPT can be used:
(ptest_sapt <- rfvimptest(data=hearth2, yname="Class", type="SAPT", Mmax=20))
ptest_sapt$testres


# If it is only of interest to test specific variables in the dataset these variables
# should be passed to rfvimptest() vias the argument 'varnames' because this
# reduces the computational burden considerably:

(ptest_twovar <- rfvimptest(data=hearth2, yname="Class", varnames=c("age", "sex"), Mmax=20))
ptest_twovar$varimp
ptest_twovar$testres


# Two-sample permutation test procedures:

# NOTE: These should be used only for informal screening for informative variables.
# They are not valid statistical tests.

# Here, the maximum number of permutations can be much higher because it is necessary
# here to construct a new forest for each permutation:
rfvimptest(data=hearth2, yname="Class", test="twosample", condinf=TRUE, Mmax=1000)

rfvimptest(data=hearth2, yname="Class", test="twosample", type="pval", condinf=TRUE, Mmax=1000)

rfvimptest(data=hearth2, yname="Class", test="twosample", type="SAPT", condinf=TRUE, Mmax=1000)

[Package rfvimptest version 0.1.3 Index]