score {scoringutils} | R Documentation |
Evaluate forecasts
Description
This function allows automatic scoring of forecasts using a range of metrics. For most users it will be the workhorse for scoring forecasts as it wraps the lower level functions package functions. However, these functions are also available if you wish to make use of them independently.
A range of forecasts formats are supported, including quantile-based,
sample-based, binary forecasts. Prior to scoring, users may wish to make use
of check_forecasts()
to ensure that the input data is in a supported
format though this will also be run internally by score()
. Examples for
each format are also provided (see the documentation for data
below or in
check_forecasts()
).
Each format has a set of required columns (see below). Additional columns may be present to indicate a grouping of forecasts. For example, we could have forecasts made by different models in various locations at different time points, each for several weeks into the future. It is important, that there are only columns present which are relevant in order to group forecasts. A combination of different columns should uniquely define the unit of a single forecast, meaning that a single forecast is defined by the values in the other columns. Adding additional unrelated columns may alter results.
To obtain a quick overview of the currently supported evaluation metrics,
have a look at the metrics data included in the package. The column
metrics$Name
gives an overview of all available metric names that can be
computed. If interested in an unsupported metric please open a feature request or consider
contributing a pull request.
For additional help and examples, check out the Getting Started Vignette as well as the paper Evaluating Forecasts with scoringutils in R.
Usage
score(data, metrics = NULL, ...)
Arguments
data |
A data.frame or data.table with the predictions and observations.
For scoring using
For scoring integer and continuous forecasts a
For scoring predictions in a quantile-format forecast you should provide
a column called
In addition a You can check the format of your data using |
metrics |
the metrics you want to have in the output. If |
... |
additional parameters passed down to |
Value
A data.table with unsummarised scores. There will be one score per
quantile or sample, which is usually not desired, so you should almost
always run summarise_scores()
on the unsummarised scores.
Author(s)
Nikos Bosse nikosbosse@gmail.com
References
Funk S, Camacho A, Kucharski AJ, Lowe R, Eggo RM, Edmunds WJ (2019) Assessing the performance of real-time epidemic forecasts: A case study of Ebola in the Western Area region of Sierra Leone, 2014-15. PLoS Comput Biol 15(2): e1006785. doi:10.1371/journal.pcbi.1006785
Examples
library(magrittr) # pipe operator
check_forecasts(example_quantile)
score(example_quantile) %>%
add_coverage(by = c("model", "target_type")) %>%
summarise_scores(by = c("model", "target_type"))
# set forecast unit manually (to avoid issues with scoringutils trying to
# determine the forecast unit automatically), check forecasts before scoring
example_quantile %>%
set_forecast_unit(
c("location", "target_end_date", "target_type", "horizon", "model")
) %>%
check_forecasts() %>%
score()
# forecast formats with different metrics
## Not run:
score(example_binary)
score(example_quantile)
score(example_integer)
score(example_continuous)
## End(Not run)
# score point forecasts (marked by 'NA' in the quantile column)
score(example_point) %>%
summarise_scores(by = "model", na.rm = TRUE)