batchmark {mlr3batchmark}

Benchmark Experiments on Batch Systems

Description

This function provides the functionality to leave the interface of mlr3 for the computation of benchmark experiments and switch over to batchtools for a more fine grained control over the execution.

batchmark() populates a batchtools::ExperimentRegistry with jobs in a mlr3::benchmark() fashion. Each combination of mlr3::Task and mlr3::Resampling defines a batchtools::Problem, each mlr3::Learner is an batchtools::Algorithm.

After the jobs have been submitted and are terminated, results can be collected with reduceResultsBatchmark() which returns a mlr3::BenchmarkResult and thus to return to the interface of mlr3.

Usage

batchmark(design, store_models = FALSE, reg = batchtools::getDefaultRegistry())

Arguments

design

(data.frame())
Data frame (or data.table::data.table()) with three columns: "task", "learner", and "resampling". Each row defines a resampling by providing a Task, Learner and an instantiated Resampling strategy. The helper function benchmark_grid() can assist in generating an exhaustive design (see examples) and instantiate the Resamplings per Task. Additionally, you can set the additional column 'param_values', see benchmark_grid().

store_models

(logical(1))
Store the fitted model in the resulting object= Set to TRUE if you want to further analyse the models or want to extract information like variable importance.

reg

batchtools::ExperimentRegistry.

Value

data.table() with ids of created jobs (invisibly).

Examples

tasks = list(mlr3::tsk("iris"), mlr3::tsk("sonar"))
learners = list(mlr3::lrn("classif.featureless"), mlr3::lrn("classif.rpart"))
resamplings = list(mlr3::rsmp("cv", folds = 3), mlr3::rsmp("holdout"))

design = mlr3::benchmark_grid(
  tasks = tasks,
  learners = learners,
  resamplings = resamplings
)

reg = batchtools::makeExperimentRegistry(NA)
batchmark(design, reg = reg)
batchtools::submitJobs(reg = reg)

reduceResultsBatchmark(reg = reg)