| cocoScore {coconots} | R Documentation |
Scoring Rule Based Model Assessment Procedure
Description
The function calculates the log, quadratic and ranked probability scores for assessing relative performance of a fitted model as proposed by Czado et al. (2009).
Usage
cocoScore(coco, val.num = 1e-10, julia = FALSE)
Arguments
coco |
An object of class coco |
val.num |
A non-negative real number which is used to stop the calculation of the score in case of GP models. The default value is 1e-10 |
julia |
if TRUE, the scores are computed with Julia. |
Details
Scoring rules assign a numerical score based on the predictive distribution and the observed data to measure the quality of probabilistic predictions. They are provided here as a model selection tool and are computed as averages over the relevant set of (in-sample) predictions. Scoring rules are, generally, negatively oriented penalties that one seeks to minimize. The literature has developed a large number of scoring rules and, unless there is a unique and clearly defined underlying decision problem, there is no automatic choice of a (proper) scoring rule to be used in any given situation. Therefore, the use of a variety of scoring rules may be appropriate to take advantage of specific emphases and strengths. Three proper scoring rules (for a definition of the concept of propriety see Gneiting and Raftery, 2007) which Jung, McCabe and Tremayne (2016) found to be particularly useful are implemented. For more information see the references listed below.
Value
a list containing the log score, quadratic score and ranked probability score.
Author(s)
Manuel Huth
References
Czado, C. and Gneitling, T. and Held, L. (2009) Predictive Model Assessment for Count Data. Biometrics, 65, 4, 1254–1261.
Gneiting, T. and Raftery, A. E. (2007) Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102:359-378.
Jung, Robert C., Brendan P. M. McCabe, and Andrew R. Tremayne. (2016). Model validation and diagnostics. In Handbook of Discrete Valued Time Series. Edited by Richard A. Davis, Scott H. Holan, Robert Lund and Nalini Ravishanker. Boca Raton: Chapman and Hall, pp. 189–218.
Jung, R. C. and Tremayne, A. R. (2011) Convolution-closed models for count timeseries with applications. Journal of Time Series Analysis, 32, 3, 268–280.
Examples
lambda <- 1
alpha <- 0.4
set.seed(12345)
data <- cocoSim(order = 1, type = "Poisson", par = c(lambda, alpha), length = 100)
#julia_installed = TRUE ensures that the fit object
#is compatible with the julia cocoScore implementation
fit <- cocoReg(order = 1, type = "Poisson", data = data)
#assessment using scoring rules - R implementation
score_r <- cocoScore(fit)