score.mvgam_forecast {mvgam}

Compute probabilistic forecast scores for mvgam objects

Description

Compute probabilistic forecast scores for mvgam objects

Usage

## S3 method for class 'mvgam_forecast'
score(
  object,
  score = "crps",
  log = FALSE,
  weights,
  interval_width = 0.9,
  n_cores = 1,
  ...
)

score(object, ...)

Arguments

object

mvgam_forecast object. See forecast.mvgam().

score

character specifying the type of proper scoring rule to use for evaluation. Options are: sis (i.e. the Scaled Interval Score), energy, variogram, elpd (i.e. the Expected log pointwise Predictive Density), drps (i.e. the Discrete Rank Probability Score) or crps (the Continuous Rank Probability Score). Note that when choosing elpd, the supplied object must have forecasts on the link scale so that expectations can be calculated prior to scoring. For all other scores, forecasts should be supplied on the response scale (i.e. posterior predictions)

log

logical. Should the forecasts and truths be logged prior to scoring? This is often appropriate for comparing performance of models when series vary in their observation ranges

weights

optional vector of weights (where length(weights) == n_series) for weighting pairwise correlations when evaluating the variogram score for multivariate forecasts. Useful for down-weighting series that have larger magnitude observations or that are of less interest when forecasting. Ignored if score != 'variogram'

interval_width

proportional value on ⁠[0.05,0.95]⁠ defining the forecast interval for calculating coverage and, if score = 'sis', for calculating the interval score

n_cores

integer specifying number of cores for calculating scores in parallel

...

Ignored

Value

a list containing scores and interval coverages per forecast horizon. If score %in% c('drps', 'crps', 'elpd'), the list will also contain return the sum of all series-level scores per horizon. If score %in% c('energy','variogram'), no series-level scores are computed and the only score returned will be for all series. For all scores apart from elpd, the in_interval column in each series-level slot is a binary indicator of whether or not the true value was within the forecast's corresponding posterior empirical quantiles. Intervals are not calculated when using elpd because forecasts will only contain the linear predictors

See Also

forecast.mvgam

Examples

# Simulate observations for three count-valued time series
data <- sim_mvgam()
# Fit a dynamic model using 'newdata' to automatically produce forecasts
mod <- mvgam(y ~ 1,
            trend_model = RW(),
            data = data$data_train,
            newdata = data$data_test,
            chains = 2)

# Extract forecasts into a 'mvgam_forecast' object
fc <- forecast(mod)

# Compute Discrete Rank Probability Scores and 0.90 interval coverages
fc_scores <- score(fc, score = 'drps')
str(fc_scores)