performance.metrics {gecko}

Performance of model predictions

Description

Calculate the performance of a model through a comparison between predicted and observed labels. Available metrics are accuracy, F1 and TSS.

Usage

performance.metrics(actual, predicted, metric)

Arguments

actual

dataframe. Same formatting as y, containg some sort of classification data.

predicted

dataframe. Same formatting as x, containg the predicted classifications of a model trained over the data in x.

metric

character. String specifying the metric used, one of accuracy, F1 and TSS.

Details

The F-score or F-measure (F1) is:

F1 = 2 \dfrac{Precision * Recall}{Precision + Recall}, with

Precision = \dfrac{True Positive}{True Positive + False Positive}

Recall = \dfrac{True Positive}{True Positive + False Negative}

Accuracy is:

\dfrac{100 * (True Postives + True Negatives)}{True Postives + True Negatives + False Positives + False Negatives}

The Pierce's skill score (PSS), Bookmaker's Informedness (BM) or True Skill Statistic (TSS) is:

TSS = TPR + TNR - 1,
with TPR being the True Positive Rate, positives correctly labelled as such and TNR, the True Negative Rate, the rate of negatives correctly labelled, such that:

TPR = \dfrac{True Positives}{True Positives + False Negatives}
TNR = \dfrac{True Negatives}{True Negatives + False Positives}
Take in consideration the fact that the F1 score is not a robust metric in datasets with class imbalances.

Value

numeric.

References

PSS: Peirce, C. S. (1884). The numerical measure of the success of predictions. Science, 4, 453–454.

Examples

observed = c("FALSE", "TRUE", "FALSE", "TRUE", "TRUE")
predicted = c("TRUE", "TRUE", "TRUE", "TRUE", "TRUE")
performance.metrics(observed, predicted, "TSS")