ensemble {autoEnsemble} | R Documentation |
Evaluate H2O Model(s) Performance
Description
Multiple model performance metrics are computed
Usage
ensemble(
models,
training_frame,
newdata = NULL,
family = "binary",
strategy = c("search"),
model_selection_criteria = c("auc", "aucpr", "mcc", "f2"),
min_improvement = 1e-05,
max = NULL,
top_rank = seq(0.01, 0.99, 0.01),
stop_rounds = 3,
reset_stop_rounds = TRUE,
stop_metric = "auc",
seed = -1,
verbatim = FALSE
)
Arguments
models |
H2O search grid or AutoML grid or a character vector of H2O model IDs.
the |
training_frame |
h2o training frame (data.frame) for model training |
newdata |
h2o frame (data.frame). the data.frame must be already uploaded on h2o server (cloud). when specified, this dataset will be used for evaluating the models. if not specified, model performance on the training dataset will be reported. |
family |
model family. currently only |
strategy |
character. the current available strategies are |
model_selection_criteria |
character, specifying the performance metrics that
should be taken into consideration for model selection. the default are
|
min_improvement |
numeric. specifies the minimum improvement in model evaluation metric to qualify further optimization search. |
max |
integer. specifies maximum number of models for each criteria to be extracted. the
default value is the |
top_rank |
numeric vector. specifies percentage of the top models taht
should be selected. if the strategy is |
stop_rounds |
integer. number of stoping rounds, in case the model stops improving |
reset_stop_rounds |
logical. if TRUE, everytime the model improves the stopping rounds penalty is resets to 0. |
stop_metric |
character. model stopping metric. the default is |
seed |
random seed (recommended) |
verbatim |
logical. if TRUE, it reports additional information about the progress of the model training, particularly used for debugging. |
Value
a matrix of F-Measures for different thresholds or the highest F-Measure value
Author(s)
E. F. Haghish
Examples
## Not run:
# load the required libraries for building the base-learners and the ensemble models
library(h2o)
library(autoEnsemble)
# initiate the h2o server
h2o.init(ignore_config = TRUE, nthreads = 2, bind_to_localhost = FALSE, insecure = TRUE)
# upload data to h2o cloud
prostate_path <- system.file("extdata", "prostate.csv", package = "h2o")
prostate <- h2o.importFile(path = prostate_path, header = TRUE)
### H2O provides 2 types of grid search for tuning the models, which are
### AutoML and Grid. Below, I tune 2 set of model grids and use them both
### for building the ensemble, just to set an example ...
#######################################################
### PREPARE AutoML Grid (takes a couple of minutes)
#######################################################
# run AutoML to tune various models (GLM, GBM, XGBoost, DRF, DeepLearning) for 120 seconds
y <- "CAPSULE"
prostate[,y] <- as.factor(prostate[,y]) #convert to factor for classification
aml <- h2o.automl(y = y, training_frame = prostate, max_runtime_secs = 120,
include_algos=c("DRF","GLM", "XGBoost", "GBM", "DeepLearning"),
# this setting ensures the models are comparable for building a meta learner
seed = 2023, nfolds = 10,
keep_cross_validation_predictions = TRUE)
#######################################################
### PREPARE H2O Grid (takes a couple of minutes)
#######################################################
# make sure equal number of "nfolds" is specified for different grids
grid <- h2o.grid(algorithm = "gbm", y = y, training_frame = prostate,
hyper_params = list(ntrees = seq(1,50,1)),
grid_id = "ensemble_grid",
# this setting ensures the models are comparable for building a meta learner
seed = 2023, fold_assignment = "Modulo", nfolds = 10,
keep_cross_validation_predictions = TRUE)
#######################################################
### PREPARE ENSEMBLE MODEL
#######################################################
### get the models' IDs from the AutoML and grid searches.
### this is all that is needed before building the ensemble,
### i.e., to specify the model IDs that should be evaluated.
ids <- c(h2o.get_ids(aml), h2o.get_ids(grid))
top <- ensemble(models = ids, training_frame = prostate, strategy = "top")
search <- ensemble(models = ids, training_frame = prostate, strategy = "search")
#######################################################
### EVALUATE THE MODELS
#######################################################
h2o.auc(aml@leader) # best model identified by h2o.automl
h2o.auc(h2o.getModel(grid@model_ids[[1]])) # best model identified by grid search
h2o.auc(top). # ensemble model with 'top' search strategy
h2o.auc(search). # ensemble model with 'search' search strategy
## End(Not run)