statistics_calibratR {CalibratR}

statistics_calibratR

Description

this method offers a variety of statistical evaluation methods for the output of the calibrate method. All returned error values represent mean error values over the n_seeds times repeated 10-fold CV.

Usage

statistics_calibratR(calibrate_object, t.test_partitions = TRUE,
  significance_models = TRUE)

Arguments

calibrate_object

list that is returned from the calibrate function. The parameter n_seeds is available as a list component of the calibrate_object

t.test_partitions

Performs a paired two sided t.test over the error values (ECE, CLE1, CLE0, MCE, AUC, sensitivity and specificity) from the random partition splits comparing a possible significant difference in mean among the calibration models. All models and the original, scaled and transformed values are tested against each other. The p_value and the effect size of the t.test are returned to the user. Can only be performed, if the calibrate_object contains a summary_CV list object, else, an error is returned. Default: TRUE

significance_models

returns important characteristics of the implemented calibration models, Default: TRUE

Details

DETAILS

Value

An object of class list, with the following components:

mean_calibration

mean of calibration error values (ECE_equal_width, MCE_equal_width, ECE_equal_freq, MCE_equal_freq, RMSE, Class 1 CLE, Class 0 CLE, Brier Score, Class 1 Brier Score, Class 0 Brier Score) over n_seeds times repeated 10-fold CV. ECE and MCE are computed once using equal-width and once using equal-frequency binning for the construction of the underlying binning scheme. Only returned, if calibrate_object contains a summary_CV list object.

standard_deviation

standard deviation of calibration error values over n_seeds times repeated 10-fold CV. Only returned, if calibrate_object contains a summary_CV list object.

var_coeff_calibration

variation coefficient of calibration error values over n_seeds times repeated 10-fold CV. Only returned, if calibrate_object contains a summary_CV list object.

mean_discrimination

mean of discrimination error (sensitivity, specificity, AUC, positive predictive value, negative predictive value, accuracy) values over n_seeds times repeated 10-fold CV. The "cut-off" is the cut-off value that maximizes sensitivity and specificity. Only returned, if calibrate_object contains a summary_CV list object.

sd_discrimination

standard deviation of discrimination error values over n_seeds times repeated 10-fold CV. Only returned, if calibrate_object contains a summary_CV list object.

var_coeff_discrimination

variation coefficient of discrimination error values over n_seeds times repeated 10-fold CV. Only returned, if calibrate_object contains a summary_CV list object.

t.test_calibration

=list(p_value=t.test.calibration, effect_size=effect_size_calibration), only returned if t.test=TRUE

t.test_discrimination

=list(p_value=t.test.discrimination, effect_size=effect_size_discrimination), only returned if t.test=TRUE

significance_models

only returned if significance_models=TRUE

n_seeds

number of random data set partitions into training and test set for folds-times CV

original_values

list object that consists of the actual and predicted values of the original scores

Author(s)

Johanna Schwarz

See Also

t.test,friedman.test

Examples

 ## Loading dataset in environment
 data(example)
 calibration_model <- example$calibration_model

 statistics <- statistics_calibratR(calibration_model)