CVST-package {CVST}R Documentation

Fast Cross-Validation via Sequential Testing

Description

The fast cross-validation via sequential testing (CVST) procedure is an improved cross-validation procedure which uses non-parametric testing coupled with sequential analysis to determine the best parameter set on linearly increasing subsets of the data. By eliminating under-performing candidates quickly and keeping promising candidates as long as possible, the method speeds up the computation while preserving the capability of a full cross-validation. Additionally to the CVST the package contains an implementation of the ordinary k-fold cross-validation with a flexible and powerful set of helper objects and methods to handle the overall model selection process. The implementations of the Cochran's Q test with permutations and the sequential testing framework of Wald are generic and can therefore also be used in other contexts.

Details

Package: CVST
Type: Package
Title: Fast Cross-Validation via Sequential Testing
Version: 0.2-3
Date: 2022-02-19
Depends: kernlab,Matrix
Author: Tammo Krueger, Mikio Braun
Maintainer: Tammo Krueger <tammokrueger@googlemail.com>
Description: The fast cross-validation via sequential testing (CVST) procedure is an improved cross-validation procedure which uses non-parametric testing coupled with sequential analysis to determine the best parameter set on linearly increasing subsets of the data. By eliminating under-performing candidates quickly and keeping promising candidates as long as possible, the method speeds up the computation while preserving the capability of a full cross-validation. Additionally to the CVST the package contains an implementation of the ordinary k-fold cross-validation with a flexible and powerful set of helper objects and methods to handle the overall model selection process. The implementations of the Cochran's Q test with permutations and the sequential testing framework of Wald are generic and can therefore also be used in other contexts.
License: GPL (>=2.0)

Index of help topics:

CV                      Perform a k-fold Cross-validation
CVST-package            Fast Cross-Validation via Sequential Testing
cochranq.test           Cochran's Q Test with Permutation
constructCVSTModel      Setup for a CVST Run.
constructData           Construction and Handling of 'CVST.data'
                        Objects
constructLearner        Construction of Specific Learners for CVST
constructParams         Construct a Grid of Parameters
constructSequentialTest
                        Construct and Handle Sequential Tests.
fastCV                  The Fast Cross-Validation via Sequential
                        Testing (CVST) Procedure
noisyDonoho             Generate Donoho's Toy Data Sets
noisySine               Regression and Classification Toy Data Set

Author(s)

Tammo Krueger, Mikio Braun

Maintainer: Tammo Krueger <tammokrueger@googlemail.com>

References

Tammo Krueger, Danny Panknin, and Mikio Braun. Fast cross-validation via sequential testing. Journal of Machine Learning Research 16 (2015) 1103-1155. URL https://jmlr.org/papers/volume16/krueger15a/krueger15a.pdf.

Abraham Wald. Sequential Analysis. Wiley, 1947.

W. G. Cochran. The comparison of percentages in matched samples. Biometrika, 37 (3-4):256–266, 1950.

M. Friedman. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32 (200):675–701, 1937.

Examples

ns = noisySine(100)
svm = constructSVMLearner()
params = constructParams(kernel="rbfdot", sigma=10^(-3:3), nu=c(0.05, 0.1, 0.2, 0.3))
opt = fastCV(ns, svm, params, constructCVSTModel())

[Package CVST version 0.2-3 Index]