comperf {gbts}

Compute model performance

Description

This function computes model performance given a vector of response values and a vector of predictions.

Usage

comperf(y, yhat, w = rep(1, length(y)), pfmc = NULL, cdfx = "fpr",
  cdfy = "tpr", dspt = 0.5)

Arguments

y

a vector of numeric response values.

yhat

a vector of model predictions.

w

an optional vector of observation weights.

pfmc

a character of the performance metric. For binary classification, pfmc accepts:

• "acc": accuracy

• "dev": deviance

• "ks": Kolmogorov-Smirnov (KS) statistic

• "auc": area under the ROC curve. Use the cdfx and cdfy arguments to specify the cumulative distributions for the x-axis and y-axis of the ROC curve, respectively. The default ROC curve is given by true positive rate (y-axis) vs. false positive rate (x-axis).

• "roc": rate on the y-axis of the ROC curve at a particular decision point (threshold) on the x-axis specified by the dspt argument. For example, if the desired performance metric is true positive rate at the 5% false positive rate, specify pfmc="roc", cdfx="fpr", cdfy="tpr", and dspt=0.05.

For regression, pfmc accepts:

• "mse": mean squared error

• "mae": mean absolute error

• "rsq": r-squared (coefficient of determination)

cdfx

a character of the cumulative distribution for the x-axis. Supported values are

• "fpr": false positive rate

• "fnr": false negative rate

• "rpp": rate of positive prediction

cdfy

a character of the cumulative distribution for the y-axis. Supported values are

• "tpr": true positive rate

• "tnr": true negative rate

dspt

a decision point (threshold) in [0, 1] for binary classification. If pfmc="acc", instances with probabilities <= dspt are predicted as negative, and those with probabilities > dspt are predicted as positive. If pfmc="roc", dspt is a threhold on the x-axis of the ROC curve such that the corresponding value on the y-axis is used as the performance metric. For example, if the desired performance metric is the true positive rate at the 5% false positive rate, specify pfmc="roc", cdfx="fpr", cdfy="tpr", and dspt=0.05.

Value

A single or a vector of numeric values of model performance, or a list of two components x and y representing the ROC curve.

Author(s)

Waley W. J. Liang <[e-mail]>

See Also

gbts, predict.gbts

Examples

y = c(0, 1, 0, 1, 1, 1)
yhat = c(0.5, 0.9, 0.2, 0.7, 0.6,  0.4)
comperf(y, yhat, pfmc = "auc")
# 0.875

y = 1:10
yhat = c(1:5 - 0.1, 6:10 + 0.1)
comperf(y, yhat, pfmc = "mse")
# 0.01