R: Model assessment based on Deviance information criterion...

gp.model.adequacy {GPBayes}

R Documentation

Model assessment based on Deviance information criterion (DIC), logarithmic pointwise predictive density (lppd), and logarithmic joint predictive density (ljpd).

Description

This function computes effective number of parameters (pD), deviance information criterion (DIC), logarithmic pointwise predictive density (lppd), and logarithmic joint predictive density (ljpd). For detailed introduction of these metrics, see Chapter 7 of Gelman et al. (2013).

The deviance function for a model with a vector of parameters \boldsymbol \theta is defined as

D(\boldsymbol \theta) = -2\log p(\mathbf{y} \mid \boldsymbol \theta),

where \mathbf{y}:=(y(\mathbf{x}_1), \ldots, y(\mathbf{x}_n))^\top is a vector of n observations.

The effective number of parameters (see p.172 of Gelman et al. 2013) is defined as

pD = E_{\boldsymbol \theta| \mathbf{y}}[D(\boldsymbol \theta)] - D(\hat{ \boldsymbol \theta }),

where \hat{\boldsymbol \theta} = E_{\boldsymbol \theta | \mathbf{y}}[\boldsymbol \theta]. The interpretation is that the effective number of parameters is the “expected" deviance minus the “fitted" deviance. Higher pD implies more over-fitting with estimate \hat{\boldsymbol \theta}.
The Deviance information criteria (DIC) (see pp. 172-173 of Gelman et al. 2013) is

DIC = E_{\boldsymbol \theta | \mathbf{y}}[D(\boldsymbol \theta)] + pD.

DIC approximates Akaike information criterion (AIC) and is more appropriate for hierarchical models than AIC and BIC.
The log predictive density (lpd) is defined as

p(y(\mathbf{x}_0) \mid \mathbf{y}) = \int p(y(\mathbf{x}_0) \mid \boldsymbol \theta, \mathbf{y}) p(\boldsymbol \theta \mid \mathbf{y}) d \boldsymbol \theta,

where \mathbf{y}:=(y(\mathbf{x}_1), \ldots, y(\mathbf{x}_n))^\top is a vector of n observations. \boldsymbol \theta contains correlation parameters and nugget parameter. This predictive density should be understood as an update of the likelihood since data is treated as prior information now. With a set of prediction locations \mathcal{X}:=\{x_0^i: i=1, \ldots, m\}. The log pointwise predictive density (lppd) is defined as

lppd = \sum_{i=1}^m \log p(y(\mathbf{x}_0^i) \mid \mathbf{y}).

The log joint predictive density (ljpd) is defined as

ljpd = \log p(y(\mathcal{X})).

The lppd is connected to cross-validation, while the ljpd measures joint uncertainty across prediction locations.

Usage

gp.model.adequacy(
  obj,
  testing.input,
  testing.output,
  pointwise = TRUE,
  joint = TRUE
)

Arguments

`obj`	a `gp` object.
`testing.input`	a matrix of testing inputs
`testing.output`	a vector of testing outputs
`pointwise`	a logical value with default value `TRUE`. If it is `TRUE`, lppd is calculated.
`joint`	a logical value with default value `TRUE`. If it is `TRUE`, ljpd is calculated.

Value

a list containing pD, DIC, lppd, ljpd.

Author(s)

Pulong Ma mpulong@gmail.com

References

Gelman, Andrew, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin (2013). Bayesian Data Analysis, Third Edition. CRC Press.