brierCurve {CSMES}

Calculates Brier Curve

Description

This function calculates the Brier curve (both in terms of cost and skew) based on a set of predictions generated by a binary classifier. Brier curves allow an evaluation of classifier performance in cost space. This code is an adapted version from the authors' original implementation, available through http://dmip.webs.upv.es/BrierCurves/BrierCurves.R.

Usage

brierCurve(labels, preds, resolution = 0.001)

Arguments

labels

Vector with true class labels

preds

Vector with predictions (real-valued or discrete)

resolution

Value for the determination of percentile intervals. Defaults to 1/1000.

Value

object of the class brierCurve which is a list with the following components:

brierCurveCost

Cost-based Brier curve, represented as (cost,loss) coordinates

brierCurveSkew

Skew-based Brier curve, represented as (skew,loss) coordinates

auc_brierCurveCost

Area under the cost-based Brier curve.

auc_brierCurveSkew

Area under the skew-based Brier curve.

Author(s)

Koen W. De Bock, [e-mail]

References

Hernandez-Orallo, J., Flach, P., & Ferri, C. (2011). Brier Curves: a New Cost-Based Visualisation of Classifier Performance. Proceedings of the 28th International Conference on Machine Learning (ICML-11), 585–592.

See Also

plotBrierCurve, CSMES.ensNomCurve

Examples

##load data
library(rpart)
data(BFP)
##generate random order vector
BFP_r<-BFP[sample(nrow(BFP),nrow(BFP)),]
size<-nrow(BFP_r)
##size<-300
train<-BFP_r[1:floor(size/3),]
val<-BFP_r[ceiling(size/3):floor(2*size/3),]
test<-BFP_r[ceiling(2*size/3):size,]
##train CART decision tree model
model=rpart(as.formula(Class~.),train,method="class")
##generate predictions for the tes set
preds<-predict(model,newdata=test)[,2]
##calculate brier curve
bc<-brierCurve(test[,"Class"],preds)