consistency {confcons}

Consistency of the predictive distribution model

Description

Calculate consistency (DCPP, DCP) of the model as the difference of the confidence calculated on the evaluation and the confidence calculated on the training subset. Consistency serves as a proxy for model's transferability.

Usage

consistency(conf_train, conf_eval)

Arguments

conf_train

Confidence calculated on the training subset: a numeric vector of length one, containing a number within the [0, 1] interval. Typically calculated by function confidence() using the training subset.

conf_eval

Confidence calculated on the evaluation subset: a numeric vector of length one, containing a number within the [0, 1] interval. Typically calculated by function confidence() using the evaluation subset.

Value

A numeric vector of length one. It is either NA_real_ or a number within the [-1, 1] interval. Typically, it falls within the [-1, 0] interval. Greater value indicates more consistent/transferable model. I.e, the closer the returned value is to -1, the less consistence/transferable the model is. Value above 0 might be an artifact or might indicate that the training and evaluation subsets were accidentally swapped.

See Also

thresholds for calculating the two thresholds, confidence for calculating confidence

Examples

# Simple examples:
consistency(conf_train = 0.93,
            conf_eval = 0.21) # -0.72 - hardly consistent/transferable model
consistency(conf_train = 0.43,
            conf_eval = 0.35) # -0.08 - consistent/transferable model, although not so confident
consistency(conf_train = 0.87,
            conf_eval = 0.71) # -0.16 - a consistent/transferable model that is confident as well
consistency(conf_train = 0.67,
            conf_eval = 0.78) # 0.11 - positive value might be an artifact
consistency(conf_train = 0.67,
            conf_eval = NA_real_) # NA

# Real-life case:
set.seed(12345)
observations <- c(rep(x = FALSE, times = 500),
                 rep(x = TRUE, times = 500))
predictions <- c(runif(n = 500, min = 0, max = 0.7),
                 runif(n = 500, min = 0.3, max = 1))
dataset <- data.frame(
	observations = observations,
	predictions = predictions,
	evaluation_mask = c(rep(x = FALSE, times = 250),
	                    rep(x = TRUE, times = 250),
	                    rep(x = FALSE, times = 250),
	                    rep(x = TRUE, times = 250))
)
thresholds_whole <- thresholds(observations = dataset$observations,
                               predictions = dataset$predictions)
confidence_training <- confidence(observations = dataset$observations[!dataset$evaluation_mask],
                                  predictions = dataset$predictions[!dataset$evaluation_mask],
                                  thresholds = thresholds_whole) # 0.602
confidence_evaluation <- confidence(observations = dataset$observations[dataset$evaluation_mask],
                                    predictions = dataset$predictions[dataset$evaluation_mask],
                                    thresholds = thresholds_whole) # 0.520
consistency(conf_train = confidence_training,
            conf_eval = confidence_evaluation) # -0.083 - consistent/transferable model

# Wrong parameterization:
try(consistency(conf_train = 1.3,
                conf_eval = 0.5)) # warning
try(consistency(conf_train = 0.6,
                conf_eval = c(0.4, 0.5))) # warning