rankWorkflows {performanceEstimation}

Provide a ranking of workflows involved in an estimation process.

Description

Given a ComparisonResults object resulting from a performance estimation experiment, this function provides a ranking (by default the top 5) of the best workflows involved in the comparison. The rankings are provided by task and for each evaluation metric.

Usage

rankWorkflows(compRes,top=min(5,length(workflowNames(compRes))),
              maxs=rep(FALSE,dim(compRes[[1]][[1]]@iterationsScores)[2]),stat="avg")

Arguments

compRes

An object of class ComparisonResults with the results of the performance estimation experiment.

top

The number of workflows to include in the rankings (defaulting to 5 or the number of workflows in the experiment if less than 5)

maxs

A vector of booleans with as many elements are there are statistics measured in the experimental comparison. A TRUE value means the respective metric is to be maximized, while a FALSE means minimization. Defaults to all FALSE values.

stat

The statistic to be used to obtain the ranks. The options are the statistics produced by the function summary applied to objects of class ComparisonResults, i.e. "avg", "std", "med", "iqr", "min", "max" or "invalid" (defaults to "avg").

Value

The function returns a named list with as many components as there are predictive tasks in the experiment. For each task you will get another named list, with as many elements as there evaluation metrics. For each of these components you have a data frame with N lines, where N is the size of the requested rank. Each line includes the name of the workflow in the respective rank position and the estimated score it got on that particular task / evaluation metric.

Author(s)

Luis Torgo [e-mail]

References

Torgo, L. (2014) An Infra-Structure for Performance Estimation and Experimental Comparison of Predictive Models in R. arXiv:1412.0436 [cs.MS] http://arxiv.org/abs/1412.0436

See Also

performanceEstimation, topPerformers, topPerformer, metricsSummary

Examples

## Not run: 
## Estimating several evaluation metrics on different variants of a
## regression tree and of a SVM, on  two data sets, using one repetition
## of  10-fold CV

data(swiss)
data(mtcars)
library(e1071)

## run the experimental comparison
results <- performanceEstimation(
               c(PredTask(Infant.Mortality ~ ., swiss),
                 PredTask(mpg ~ ., mtcars)),
               c(workflowVariants(learner='svm',
                                  learner.pars=list(cost=c(1,5),gamma=c(0.1,0.01))
                                 )
               ),
               EstimationTask(metrics=c("mse","mae"),method=CV(nReps=2,nFolds=5))
                                 )
## get a ranking of the top workflows for each task and evaluation metric
rankWorkflows(results)
## get a ranking of the top workflows for each task and evaluation
## metric by the median score on all iterations instead of the mean score
rankWorkflows(results, stat="med")

## End(Not run)