valstats {CGGP}

Calculate stats for prediction on validation data

Description

Calculate stats for prediction on validation data

Usage

valstats(
  predmean,
  predvar,
  Yval,
  bydim = TRUE,
  RMSE = TRUE,
  score = TRUE,
  CRPscore = TRUE,
  coverage = TRUE,
  corr = TRUE,
  R2 = TRUE,
  MAE = FALSE,
  MIS90 = FALSE,
  metrics,
  min_var = .Machine$double.eps
)

Arguments

predmean

Predicted mean

predvar

Predicted variance

Yval

Y validation data

bydim

If multiple outputs, should it be done separately by dimension?

RMSE

Should root mean squared error (RMSE) be included?

score

Should score be included?

CRPscore

Should CRP score be included?

coverage

Should coverage be included?

corr

Should correlation between predicted and true mean be included?

R2

Should R^2 be included?

MAE

Should mean absolute error (MAE) be included?

MIS90

Should mean interval score for 90% confidence be included? See Gneiting and Raftery (2007).

metrics

Optional additional metrics to be calculated. Should have same first three parameters as this function.

min_var

Minimum value of the predicted variance. Negative or zero variances can cause errors.

Value

data frame

References

Gneiting, Tilmann, and Adrian E. Raftery. "Strictly proper scoring rules, prediction, and estimation." Journal of the American Statistical Association 102.477 (2007): 359-378.

Examples

valstats(c(0,1,2), c(.01,.01,.01), c(0,1.1,1.9))
valstats(cbind(c(0,1,2), c(1,2,3)),
         cbind(c(.01,.01,.01),c(.1,.1,.1)),
         cbind(c(0,1.1,1.9),c(1,2,3)))
valstats(cbind(c(0,1,2), c(8,12,34)),
         cbind(c(.01,.01,.01),c(1.1,.81,1.1)),
         cbind(c(0,1.1,1.9),c(10,20,30)), bydim=FALSE)
valstats(cbind(c(.8,1.2,3.4), c(8,12,34)),
         cbind(c(.01,.01,.01),c(1.1,.81,1.1)),
         cbind(c(1,2,3),c(10,20,30)), bydim=FALSE)