scores_sample_univ {scoringRules}

Scoring Rules for Simulated Forecast Distributions

Description

Calculate scores (CRPS, LogS, DSS) given observations and draws from the predictive distributions.

Usage

crps_sample(
  y,
  dat,
  method = "edf",
  w = NULL,
  bw = NULL,
  num_int = FALSE,
  show_messages = TRUE
)

logs_sample(y, dat, bw = NULL, show_messages = FALSE)

dss_sample(y, dat, w = NULL)

Arguments

y

vector of realized values.

dat

vector or matrix (depending on y; see details) of simulation draws from forecast distribution.

method

string; approximation method. Options: "edf" (empirical distribution function) and "kde" (kernel density estimation).

w

optional; vector or matrix (matching dat) of weights for method "edf".

bw

optional; vector (matching y) of bandwidths for kernel density estimation; see details.

num_int

logical; if TRUE numerical integration is used for method "kde".

show_messages

logical; display of messages (does not affect warnings and errors).

Details

For a vector y of length n, dat should be given as a matrix with n rows. If y has length 1, then dat may be a vector.

crps_sample employs an empirical version of the quantile decomposition of the CRPS (Laio and Tamea, 2007) when using method = "edf". For method = "kde", it uses kernel density estimation using a Gaussian kernel. The logarithmic score always uses kernel density estimation.

The bandwidth (bw) for kernel density estimation can be specified manually, in which case it must be a positive number. If bw == NULL, the bandwidth is selected using the core function bw.nrd. Numerical integration may speed up computation for crps_sample in case of large samples dat.

Value

Value of the score. A lower score indicates a better forecast.

Author(s)

Alexander Jordan, Fabian Krueger, Sebastian Lerch

References

Evaluating simulation based forecast distributions:

Krueger, F., Lerch, S., Thorarinsdottir, T.L. and T. Gneiting (2021): ‘Predictive inference based on Markov chain Monte Carlo output’, International Statistical Review 89, 274-301. doi:10.1111/insr.12405

Empirical quantile decomposition of the CRPS:

Laio, F. and S. Tamea (2007): 'Verification tools for probabilistic forecasts of continuous hydrological variables', Hydrology and Earth System Sciences, 11, 1267-1277. doi:10.5194/hess-11-1267-2007

See Also

scores_sample_univ_weighted for weighted versions of the scoring rules documented here.

Examples

## Not run: 

# y has length greater than 1
y <- 1:2
sample <- matrix(rnorm(20), nrow = 2)
crps_sample(y = y, dat = sample)
logs_sample(y = y, dat = sample)

y <- 1:2
sample <- rnorm(10)
crps_sample(y = y, dat = sample) # error

# y has length 1
y <- 1
sample <- rnorm(10)
crps_sample(y = y, dat = sample)

sample <- matrix(rnorm(10), nrow = 1)
crps_sample(y = y, dat = sample)

sample <- matrix(rnorm(20), nrow = 2)
crps_sample(y = y, dat = sample) # error

## End(Not run)