RandomWalkTest {s2dv}

Random Walk test for skill differences

Description

Forecast comparison of the skill obtained with 2 forecasts (with respect to a common observational reference) based on Random Walks (DelSole and Tippett, 2016).

Usage

RandomWalkTest(
  skill_A,
  skill_B,
  time_dim = "sdate",
  test.type = "two.sided.approx",
  alpha = 0.05,
  pval = TRUE,
  sign = FALSE,
  ncores = NULL
)

Arguments

skill_A

A numerical array of the time series of the scores obtained with the forecaster A.

skill_B

A numerical array of the time series of the scores obtained with the forecaster B. The dimensions should be identical as parameter 'skill_A'.

time_dim

A character string indicating the name of the dimension along which the tests are computed. The default value is 'sdate'.

test.type

A character string indicating the type of significance test. It can be "two.sided.approx" (to assess whether forecaster A and forecaster B are significantly different in terms of skill with a two-sided test using the approximation of DelSole and Tippett, 2016), "two.sided" (to assess whether forecaster A and forecaster B are significantly different in terms of skill with an exact two-sided test), "greater" (to assess whether forecaster A shows significantly better skill than forecaster B with a one-sided test for negatively oriented scores), or "less" (to assess whether forecaster A shows significantly better skill than forecaster B with a one-sided test for positively oriented scores). The default value is "two.sided.approx".

alpha

A numeric of the significance level to be used in the statistical significance test (output "sign"). The default value is 0.05.

pval

A logical value indicating whether to return the p-value of the significance test. The default value is TRUE.

sign

A logical value indicating whether to return the statistical significance of the test based on 'alpha'. The default value is FALSE.

ncores

An integer indicating the number of cores to use for parallel computation. The default value is NULL.

Details

Null and alternative hypothesis for "two-sided" test (regardless of the orientation of the scores):
H0: forecaster A and forecaster B are not different in terms of skill
H1: forecaster A and forecaster B are different in terms of skill

Null and alternative hypothesis for one-sided "greater" (for negatively oriented scores, i.e., the lower the better) and "less" (for positively oriented scores, i.e., the higher the better) tests:
H0: forecaster A is not better than forecaster B
H1: forecaster A is better than forecaster B

Examples of negatively oriented scores are the RPS, RMSE and the Error, while the ROC score is a positively oriented score.

DelSole and Tippett (2016) approximation for two-sided test at 95 level: significant if the difference between the number of times that forecaster A has been better than forecaster B and forecaster B has been better than forecaster A is above 2sqrt(N) or below -2sqrt(N).

Value

A list with:

$score

A numerical array with the same dimensions as the input arrays except 'time_dim'. The number of times that forecaster A has been better than forecaster B minus the number of times that forecaster B has been better than forecaster A (for skill negatively oriented, i.e., the lower the better). If $score is positive, forecaster A has been better more times than forecaster B. If $score is negative, forecaster B has been better more times than forecaster A.

$sign

A logical array of the statistical significance with the same dimensions as the input arrays except "time_dim". Returned only if "sign" is TRUE.

$p.val

A numeric array of the p-values with the same dimensions as the input arrays except "time_dim". Returned only if "pval" is TRUE.

References

DelSole and Tippett (2016): https://doi.org/10.1175/MWR-D-15-0218.1

Examples

fcst_A <- array(data = 11:50, dim = c(sdate = 10, lat = 2, lon = 2))
fcst_B <- array(data = 21:60, dim = c(sdate = 10, lat = 2, lon = 2))
reference <- array(data = 1:40, dim = c(sdate = 10, lat = 2, lon = 2))
scores_A <- abs(fcst_A - reference)
scores_B <- abs(fcst_B - reference)
res1 <- RandomWalkTest(skill_A = scores_A, skill_B = scores_B, pval = FALSE, sign = TRUE)
res2 <- RandomWalkTest(skill_A = scores_A, skill_B = scores_B, test.type = 'greater')