REL {SeaVal}

Reliability score

Description

Computes both the reliability component of the Brier score or reliability component of the Ignorance score. Mason claims to prefer the ignorance score version, but this has a very high chance of being NA. Mason writes that the scores are unstable for single locations and that one should pool over many locations. Requires the specification of probability bins. One score for each category (below, normal, above) and also the sum of the scores.

Values close to 0 indicate reliable forecasts. Higher values mean less reliable forecasts.

Usage

REL(
  dt,
  bins = c(0.3, 0.35001),
  f = c("below", "normal", "above"),
  o = tc_cols(dt),
  by = by_cols_terc_fc_score(),
  pool = "year",
  dim.check = TRUE
)

Arguments

dt

Data table containing the predictions.

bins

probability bins, defaults to ("<30", "30-35",">35") which is given as c(0.30, 0.35001).

f

column names of the prediction.

o

column name of the observations (either in obs_dt, or in dt if obs_dt = NULL). The observation column needs to contain -1 if it falls into the first category (corresponding to fcs[1]), 0 for the second and 1 for the third category.

by

column names of grouping variables, all of which need to be columns in dt. Default is to group by all instances of month, season, lon, lat, system and lead_time that are columns in dt.

pool

column name(s) for the variable(s) along which is averaged, typically just 'year'.

dim.check

Logical. If TRUE, the function tests whether the data table contains only one row per coordinate-level, as should be the case.

Value

A data table with the scores

Examples

dt = data.table(below = c(0.5,0.3,0),
                normal = c(0.3,0.3,0.7),
                above = c(0.2,0.4,0.3),
                tc_cat = c(-1,0,0),
                year = 1:3)
print(dt)
REL(dt)