brierscore {scoring} | R Documentation |
Calculate Brier Scores And Decompositions
Description
Calculate Brier scores, average Brier scores by a grouping variable, and Brier score decompositions for two-alternative forecasts.
Usage
brierscore(object, data, group = NULL, decomp = FALSE, bounds = NULL,
reverse = FALSE, wt = NULL, decompControl = list())
Arguments
object |
an object of class "formula", of the form
|
data |
an optional data frame or list containing the
variables in the formula. If not found in |
group |
the name of a grouping variable within |
decomp |
if |
bounds |
a vector of length 2 corresponding to the desired minimum and maximum Brier score, respectively. |
reverse |
if |
wt |
a vector of weights for computing a weighted Brier score. If |
decompControl |
a list of additional settings for the Brier decomposition. See options below. |
Details
If decomp=TRUE
or group
is supplied, the function returns
a list (see value section). Otherwise, the function returns a numeric
vector containing the Brier score associated with each forecast.
Some decompControl
arguments are specifically designed for forecasting tournaments and may not be useful in other situations. Possible arguments for decompControl
include:
- wt
A vector of weights, for performing a weighted Brier decomposition (could also use the simple
wt
argument).- qid
A vector of question ids, for use with the
qtype
argument.- bin
If
TRUE
(default), forecasts are binned prior to decomposition. IfFALSE
, the original forecasts are maintained.- qtype
A data frame with columns
qid
,ord
,squo
. For each unique question id in theqid
argument above, this describes whether or not the question is ordinal (1=yes,0=no) and whether or not the question has a "status quo" interpretation (1=yes,0=no).- scale
Should Brier components be rescaled, such that 1 is always best and 0 is always worst? Defaults to
FALSE
.- roundto
To what value should forecasts be rounded (necessary for Murphy decomposition)? Defaults to .1, meaning that forecasts are rounded to the nearest .1.
- binstyle
Method for ensuring that each forecast sums to 1. If equal to 1 (default), the smallest forecast is one minus the sum of the other forecasts. If equal to 2, the forecast furthest from its rounded value is one minus the sum of other forecasts.
- resamples
Desired number of Brier resamples (useful for questions with inconsistent alternatives). Defaults to 0; see Merkle & Hartman reference for more detail.
Value
Depending on input arguments, brierscore
may return an object of
class numeric
containing raw Brier scores. It may also return
a list containing the objects below.
rawscores |
an object of class |
brieravg |
an object of class |
decomp |
an object of class |
Author(s)
Ed Merkle
References
Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1-3.
Merkle, E. C. & Hartman, R. (2018). Weighted Brier score decompositions for topically heterogenous forecasting tournaments. Working paper.
Murphy, A. H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology, 12, 595-600.
Yates, J. F. (1982). External correspondence: Decompositions of the mean probability score. Organizational Behavior and Human Performance, 30, 132-156.
Young, R. M. B. (2010). Decomposition of the Brier score for weighted forecast-verification pairs. Quarterly Journal of the Royal Meteorological Society, 136, 1364-1370.
See Also
Examples
data("WorldEvents")
## Raw Brier scores
brier1 <- brierscore(answer ~ forecast, data=WorldEvents)
## Raw Brier scores plus group means and decompositions
brier2 <- brierscore(answer ~ forecast, data=WorldEvents,
group="forecaster", decomp=TRUE)
## Obtain Brier scores via calcscore
brier3 <- calcscore(answer ~ forecast, data=WorldEvents,
param=2, fam="pow")
all.equal(brier1, brier3)