quantile_score {scoringutils}

Quantile Score

Description

Proper Scoring Rule to score quantile predictions. Smaller values are better. The quantile score is closely related to the Interval score (see interval_score()) and is the quantile equivalent that works with single quantiles instead of central prediction intervals.

Usage

quantile_score(true_values, predictions, quantiles, weigh = TRUE)

Arguments

true_values

A vector with the true observed values of size n

predictions

nxN matrix of predictive samples, n (number of rows) being the number of data points and N (number of columns) the number of Monte Carlo samples. Alternatively, predictions can just be a vector of size n.

quantiles

vector of size n with the quantile values of the corresponding predictions.

weigh

if TRUE, weigh the score by alpha / 2, so it can be averaged into an interval score that, in the limit, corresponds to CRPS. Alpha is the value that corresponds to the (alpha/2) or (1 - alpha/2) quantiles provided and will be computed from the quantile. Alpha is the decimal value that represents how much is outside a central prediction interval (E.g. for a 90 percent central prediction interval, alpha is 0.1). Default: TRUE.

Value

vector with the scoring values

References

Strictly Proper Scoring Rules, Prediction,and Estimation, Tilmann Gneiting and Adrian E. Raftery, 2007, Journal of the American Statistical Association, Volume 102, 2007 - Issue 477

Evaluating epidemic forecasts in an interval format, Johannes Bracher, Evan L. Ray, Tilmann Gneiting and Nicholas G. Reich, https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1008618

Examples

true_values <- rnorm(10, mean = 1:10)
alpha <- 0.5

lower <- qnorm(alpha / 2, rnorm(10, mean = 1:10))
upper <- qnorm((1 - alpha / 2), rnorm(10, mean = 1:10))

qs_lower <- quantile_score(true_values,
  predictions = lower,
  quantiles = alpha / 2
)
qs_upper <- quantile_score(true_values,
  predictions = upper,
  quantiles = 1 - alpha / 2
)
interval_score <- (qs_lower + qs_upper) / 2