metrics {mixvlmc}

Predictive quality metrics for context based models

Description

This function computes and returns predictive quality metrics for context based models such as VLMC and VLMC with covariates.

Usage

metrics(model, ...)

Arguments

model

The context based model on which to compute predictive metrics.

...

Additional parameters for predictive metrics computation.

Details

A context based model computes transition probabilities for its contexts. Using a maximum transition probability decision rule, this can be used to predict the new state that is the more likely to follow the current one, given the context (see predict.vlmc()). The quality of these predictions is evaluated using standard metrics including:

• accuracy

• the full confusion matrix

• the area under the roc curve (AUC), considering the context based model as a (conditional) probability estimator. We use Hand and Till (2001) multiclass AUC in case of a state space with more than 2 states

Value

The returned value is guaranteed to have at least three components

• accuracy: the accuracy of the predictions

• conf_mat: the confusion matrix of the predictions, with predicted values in rows and true values in columns

• auc: the AUC of the predictive model

References

David J. Hand and Robert J. Till (2001). "A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems." Machine Learning 45(2), p. 171–186. DOI: doi:10.1023/A:1010920819831.

See Also

metrics.vlmc(), metrics.ctx_node(), contexts.vlmc(), predict.vlmc().

Examples

pc <- powerconsumption[powerconsumption$week == 5, ]
breaks <- c(
  0,
  median(powerconsumption$active_power, na.rm = TRUE),
  max(powerconsumption$active_power, na.rm = TRUE)
)
labels <- c(0, 1)
dts <- cut(pc$active_power, breaks = breaks, labels = labels)
model <- vlmc(dts)
metrics(model)