fineTuning,knnFineTune {regtools}

Grid Search Plus More

Description

Adds various extra features to grid search for specified tuning parameter/hyperparameter combinations: There is a plot() function, using parallel coordinates graphs to show trends among the different combinations; and Bonferroni confidence intervals are computed to avoid p-hacking. An experimental smoothing facility is also included.

Usage

fineTuning(dataset,pars,regCall,nCombs=NULL,specCombs=NULL,nTst=500,
   nXval=1,up=TRUE,k=NULL,dispOrderSmoothed=FALSE,
   showProgress=TRUE,...)
## S3 method for class 'tuner'
plot(x,...)
knnFineTune(data,yName,k,expandVars,ws,classif=FALSE,seed=9999)
fineTuningPar(cls,dataset,pars,regCall,nCombs=NULL,specCombs=NULL,
   nTst=500,nXval=1,up=TRUE,k=NULL,dispOrderSmoothed=FALSE)

Arguments

...

Arguments to be passed on by fineTuning or plot.tuner.

x

Output object from fineTuning.

cls

A parallel cluster.

dataset

Data frame etc. containing the data to be analyzed.

data

The data to be analyzed.

yName

Quoted name of "Y" in the column names of data.

expandVars

Indices of columns in data to be weighted in distance calculations.

ws

Weights to be used for expandVars.

classif

Set to TRUE for classification problems.

seed

Seed for random number generation.

pars

R list, showing the desired tuning parameter values.

regCall

Function to be called at each parameter combination, performing the model fit etc.

nCombs

Number of parameter combinations to run. If Null, all will be run

.

nTst

Number of data points to be in the test set.

nXval

Number of folds to be run for a given data partition and parameter combination.

k

Nearest-neighbor smoothing parameter.

up

If TRUE, display results in ascending order of performance value.

dispOrderSmoothed

Display in order of smoothed results.

showProgress

If TRUE, print each output line as it becomes ready.

specCombs

A data frame in which the user specifies # hyperparameter parameter combinations to evaluate.

Details

The user specifies the values for each tuning parameter in pars. This leads to a number of possible combinations of the parameters. In many cases, there are more combinations than the user wishes to try, so nCombs of them will be chosen at random.

For each combination, the function will run the analysis specified by the user in regCall. The latter must have the call form

ftnName(dtrn,dtst,cmbi

Again, note that it is fineTuning that calls this function. It will provide the training and test sets dtrn and dtst, as well as cmbi ("combination i"), the particular parameter combination to be run at this moment.

Each chosen combination is run in nXval folds. All specified combinations are run fully, as opposed to a directional "hill descent" search that hopes it might eliminate poor combinations early in the process.

The function knnFineTune is a wrapper for fineTuning for k-NN problems.

The function plot.tuner draws a parallel coordinates plot to visualize the grid. The argument x is the output of fineTuning. Arguments to specify in the ellipsis are: col is the column to be plotted; disp is the number to display, with 0, -m and +m meaning cases with the m smallest 'smoothed' values, all cases and the m largest values of 'smoothed', respectively; jit avoids plotting coincident lines by adding jitter in the amount jit * range(x) * runif(n,-0.5,0.5).

Value

Object of class **”tuner'**. Contains the grid results, including upper bounds of approximate one-sided 95 univariate and Bonferroni-Dunn (adjusted for the number of parameter combinations).

Author(s)

Norm Matloff

Examples

# mlb data set, predict weight using k-NN, try various values of k

tc <- function(dtrn,dtst,cmbi,...)
{
   knnout <- kNN(dtrn[,-3],dtrn[,3],dtst[,-3],as.integer(cmbi[1]))
   preds <- knnout$regests
   mean(abs(preds - dtst[,3]))
}

data(mlb)
mlb <- mlb[,3:6]
mlb.d <- factorsToDummies(mlb)
fineTuning(mlb.d,list(k=c(5,25)),tc,nTst=100,nXval=2)