ArchiveAsyncTuning {mlr3tuning}

Rush Data Storage

Description

The 'ArchiveAsyncTuning“ stores all evaluated hyperparameter configurations and performance scores in a rush::Rush database.

Details

The ArchiveAsyncTuning is a connector to a rush::Rush database.

Data Structure

The table (⁠$data⁠) has the following columns:

• One column for each hyperparameter of the search space (⁠$search_space⁠).

• One (list-)column for the internal_tuned_values

• One column for each performance measure (⁠$codomain⁠).

• x_domain (list())
  Lists of (transformed) hyperparameter values that are passed to the learner.

• runtime_learners (numeric(1))
  Sum of training and predict times logged in learners per mlr3::ResampleResult / evaluation. This does not include potential overhead time.

• timestamp (POSIXct)
  Time stamp when the evaluation was logged into the archive.

• batch_nr (integer(1))
  Hyperparameters are evaluated in batches. Each batch has a unique batch number.

Analysis

For analyzing the tuning results, it is recommended to pass the ArchiveAsyncTuning to as.data.table(). The returned data table contains the mlr3::ResampleResult for each hyperparameter evaluation.

S3 Methods

• as.data.table.ArchiveTuning(x, unnest = "x_domain", exclude_columns = "uhash", measures = NULL)
  Returns a tabular view of all evaluated hyperparameter configurations.
  ArchiveAsyncTuning -> data.table::data.table()
  

  • x (ArchiveAsyncTuning)

  • unnest (character())
    Transforms list columns to separate columns. Set to NULL if no column should be unnested.

  • exclude_columns (character())
    Exclude columns from table. Set to NULL if no column should be excluded.

  • measures (List of mlr3::Measure)
    Score hyperparameter configurations on additional measures.

Super classes

bbotk::Archive -> bbotk::ArchiveAsync -> ArchiveAsyncTuning

Active bindings

internal_search_space

(paradox::ParamSet)
The search space containing those parameters that are internally optimized by the mlr3::Learner.

benchmark_result

(mlr3::BenchmarkResult)
Benchmark result.

Methods

Public methods

• ArchiveAsyncTuning$new()

• ArchiveAsyncTuning$learner()

• ArchiveAsyncTuning$learners()

• ArchiveAsyncTuning$learner_param_vals()

• ArchiveAsyncTuning$predictions()

• ArchiveAsyncTuning$resample_result()

• ArchiveAsyncTuning$print()

• ArchiveAsyncTuning$clone()

Inherited methods

• bbotk::Archive$format()
• bbotk::Archive$help()
• bbotk::ArchiveAsync$best()
• bbotk::ArchiveAsync$clear()
• bbotk::ArchiveAsync$data_with_state()
• bbotk::ArchiveAsync$nds_selection()
• bbotk::ArchiveAsync$pop_point()
• bbotk::ArchiveAsync$push_failed_point()
• bbotk::ArchiveAsync$push_points()
• bbotk::ArchiveAsync$push_result()
• bbotk::ArchiveAsync$push_running_point()

Method new()

Creates a new instance of this R6 class.

Usage

ArchiveAsyncTuning$new(
  search_space,
  codomain,
  rush,
  internal_search_space = NULL
)

Arguments

search_space

(paradox::ParamSet)
Hyperparameter search space. If NULL (default), the search space is constructed from the paradox::TuneToken of the learner's parameter set (learner$param_set).

codomain

(bbotk::Codomain)
Specifies codomain of objective function i.e. a set of performance measures. Internally created from provided mlr3::Measures.

rush

(Rush)
If a rush instance is supplied, the tuning runs without batches.

internal_search_space

(paradox::ParamSet or NULL)
The internal search space of the tuner. This includes parameters that the learner can optimize internally durign ⁠$train()⁠, such as the number of epochs via early stopping.

check_values

(logical(1))
If TRUE (default), hyperparameter configurations are check for validity.

Method learner()

Retrieve mlr3::Learner of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive. Learner does not contain a model. Use ⁠$learners()⁠ to get learners with models.

Usage

ArchiveAsyncTuning$learner(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(logical(1))
The uhash value to filter for.

Method learners()

Retrieve list of trained mlr3::Learner objects of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveAsyncTuning$learners(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(logical(1))
The uhash value to filter for.

Method learner_param_vals()

Retrieve param values of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveAsyncTuning$learner_param_vals(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(logical(1))
The uhash value to filter for.

Method predictions()

Retrieve list of mlr3::Prediction objects of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveAsyncTuning$predictions(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(logical(1))
The uhash value to filter for.

Method resample_result()

Retrieve mlr3::ResampleResult of the i-th evaluation, by position or by unique hash uhash. i and uhash are mutually exclusive.

Usage

ArchiveAsyncTuning$resample_result(i = NULL, uhash = NULL)

Arguments

i

(integer(1))
The iteration value to filter for.

uhash

(logical(1))
The uhash value to filter for.

Method print()

Printer.

Usage

ArchiveAsyncTuning$print()

Arguments

...

(ignored).

Method clone()

The objects of this class are cloneable with this method.

Usage

ArchiveAsyncTuning$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.