baseline_binomial {cvms}  R Documentation 
Create a baseline evaluation of a test set.
In modelling, a baseline is a result that
is meaningful to compare the results from our models to. For instance, in
classification, we usually want our results to be better than random guessing.
E.g. if we have three classes, we can expect an accuracy of 33.33%
, as for every
observation we have 1/3
chance of guessing the correct class. So our model should achieve
a higher accuracy than 33.33%
before it is more useful to us than guessing.
While this expected value is often fairly straightforward to find analytically, it
only represents what we can expect on average. In reality, it's possible to get far better
results than that by guessing.
baseline_binomial()
finds the range of likely values by evaluating multiple sets
of random predictions and summarizing them with a set of useful descriptors. Additionally,
it evaluates a set of all 0
predictions and
a set of all 1
predictions.
baseline_binomial(
test_data,
dependent_col,
n = 100,
metrics = list(),
positive = 2,
cutoff = 0.5,
parallel = FALSE
)
test_data 

dependent_col 
Name of dependent variable in the supplied test and training sets. 
n 
The number of sets of random predictions to evaluate. (Default is 
metrics 
E.g. You can enable/disable all metrics at once by including
The Also accepts the string 
positive 
Level from dependent variable to predict.
Either as character (preferable) or level index ( E.g. if we have the levels Note: For reproducibility, it's preferable to specify the name directly, as
different Used when calculating confusion matrix metrics and creating N.B. Only affects evaluation metrics, not the returned predictions. 
cutoff 
Threshold for predicted classes. (Numeric) 
parallel 
Whether to run the Remember to register a parallel backend first.
E.g. with 
Packages used:
ROC
and AUC
: pROC::roc
list
containing:
a tibble
with summarized results (called summarized_metrics
)
a tibble
with random evaluations (random_evaluations
)
....................................................................
Based on the generated test set predictions,
a confusion matrix and ROC
curve are used to get the following:
ROC
:
AUC
, Lower CI
, and Upper CI
Note, that the ROC
curve is only computed when AUC
is enabled.
Confusion Matrix
:
Balanced Accuracy
,
Accuracy
,
F1
,
Sensitivity
, Specificity
,
Positive Predictive Value
,
Negative Predictive Value
,
Kappa
,
Detection Rate
,
Detection Prevalence
,
Prevalence
, and
MCC
(Matthews correlation coefficient).
....................................................................
The Summarized Results tibble
contains:
The Measure column indicates the statistical descriptor used on the evaluations.
The row where Measure == All_0
is the evaluation when all predictions are 0
.
The row where Measure == All_1
is the evaluation when all predictions are 1
.
The aggregated metrics.
....................................................................
The Random Evaluations tibble
contains:
The nonaggregated metrics.
A nested tibble
with the predictions and targets.
A list
of ROC curve objects (if computed).
A nested tibble
with the confusion matrix.
The Pos_
columns tells you whether a row is a
True Positive (TP
), True Negative (TN
), False Positive (FP
),
or False Negative (FN
), depending on which level is the "positive" class.
I.e. the level you wish to predict.
A nested Process information object with information about the evaluation.
Name of dependent variable.
Ludvig Renbo Olsen, rpkgs@ludvigolsen.dk
Other baseline functions:
baseline_gaussian()
,
baseline_multinomial()
,
baseline()
# Attach packages
library(cvms)
library(groupdata2) # partition()
library(dplyr) # %>% arrange()
# Data is part of cvms
data < participant.scores
# Set seed for reproducibility
set.seed(1)
# Partition data
partitions < partition(data, p = 0.7, list_out = TRUE)
train_set < partitions[[1]]
test_set < partitions[[2]]
# Create baseline evaluations
# Note: usually n=100 is a good setting
baseline_binomial(
test_data = test_set,
dependent_col = "diagnosis",
n = 2
)
# Parallelize evaluations
# Attach doParallel and register four cores
# Uncomment:
# library(doParallel)
# registerDoParallel(4)
# Make sure to uncomment the parallel argument
baseline_binomial(
test_data = test_set,
dependent_col = "diagnosis",
n = 4
#, parallel = TRUE # Uncomment
)