R: Perform leave-one-out (loo) cross-validation

loo_validation.bgmfit {bsitar}

R Documentation

Perform leave-one-out (loo) cross-validation

Description

The loo_validation() is a wrapper around the brms::loo() function to perform approximate leave-one-out cross-validation based on the posterior likelihood. See brms::loo() for more details.

Usage

## S3 method for class 'bgmfit'
loo_validation(
  model,
  compare = TRUE,
  resp = NULL,
  pointwise = FALSE,
  moment_match = FALSE,
  reloo = FALSE,
  k_threshold = 0.7,
  save_psis = FALSE,
  moment_match_args = list(),
  reloo_args = list(),
  model_names = NULL,
  ndraws = NULL,
  draw_ids = NULL,
  cores = 1,
  deriv_model = NULL,
  verbose = FALSE,
  dummy_to_factor = NULL,
  expose_function = FALSE,
  usesavedfuns = NULL,
  clearenvfuns = NULL,
  envir = NULL,
  ...
)

loo_validation(model, ...)

Arguments

`model`	An object of class `bgmfit`.
`compare`	A flag indicating if the information criteria of the models should be compared to each other via `loo::loo_compare()`.
`resp`	Optional names of response variables. If specified, predictions are performed only for the specified response variables.
`pointwise`	A flag indicating whether to compute the full log-likelihood matrix at once or separately for each observation. The latter approach is usually considerably slower but requires much less working memory. Accordingly, if one runs into memory issues, `pointwise = TRUE` is the way to go.
`moment_match`	A logical argument to indicate whether `loo::loo_moment_match()` should be applied on problematic observations. Defaults to `FALSE`. For most models, moment matching will only work if you have set `save_pars = save_pars(all = TRUE)` when fitting the model with `brms::brm()`. See `brms::loo_moment_match()` for more details.
`reloo`	A logical argument to indicate whether `brms::reloo()` should be applied on problematic observations. Defaults to `FALSE`.
`k_threshold`	The threshold at which pareto `k` estimates are treated as problematic. Defaults to `0.7`. Only used if argument `reloo` is `TRUE`. See `pareto_k_ids` for more details.
`save_psis`	Should the `"psis"` object created internally be saved in the returned object? For more details see `loo`.
`moment_match_args`	An optional `list` of additional arguments passed to `loo::loo_moment_match()`.
`reloo_args`	An optional `list` of additional arguments passed to `brms::reloo()`.
`model_names`	If `NULL` (the default) will use model names derived from deparsing the call. Otherwise will use the passed values as model names.
`ndraws`	A positive integer indicating the number of posterior draws to be used in estimation. If `NULL` (default), all draws are used.
`draw_ids`	An integer indicating the specific posterior draw(s) to be used in estimation (default `NULL`).
`cores`	Number of cores to be used when running the parallel computations (if `future = TRUE`). On non-Windows systems this argument can be set globally via the mc.cores option. For the default `NULL` option, the number of cores are set automatically by calling the `future::availableCores()`. The number of cores used are the maximum number of cores avaialble minus one, i.e., `future::availableCores() - 1`.
`deriv_model`	A logical to specify whether to estimate velocity curve from the derivative function, or the differentiation of the distance curve. The argument `deriv_model` is set to `TRUE` for those functions which need velocity curve such as `growthparameters()` and `plot_curves()`, and `NULL` for functions which explicitly use the distance curve (i.e., fitted values) such as `loo_validation()` and `plot_ppc()`.
`verbose`	An optional argument (logical, default `FALSE`) to indicate whether to print information collected during setting up the object(s).
`dummy_to_factor`	A named list (default `NULL`) that is used to convert dummy variables into a factor variable. The named elements are `factor.dummy`, `factor.name`, and `factor.level`. The `factor.dummy` is a vector of character strings that need to be converted to a factor variable whereas the `factor.name` is a single character string that is used to name the newly created factor variable. The `factor.level` is used to name the levels of newly created factor. When `factor.name` is `NULL`, then the factor name is internally set as `'factor.var'`. If `factor.level` is `NULL`, then names of factor levels are take from the `factor.dummy` i.e., the factor levels are assigned same name as `factor.dummy`. Note that when `factor.level` is not `NULL`, its length must be same as the length of the `factor.dummy`.
`expose_function`	An optional logical argument to indicate whether to expose Stan functions (default `FALSE`). Note that if user has already exposed Stan functions during model fit by setting `expose_function = TRUE` in the `bsitar()`, then those exposed functions are saved and can be used during post processing of the posterior draws and therefore `expose_function` is by default set as `FALSE` in all post processing functions except `optimize_model()`. For `optimize_model()`, the default setting is `expose_function = NULL`. The reason is that each optimized model has different Stan function and therefore it need to be re exposed and saved. The `expose_function = NULL` implies that the setting for `expose_function` is taken from the original `model` fit. Note that `expose_function` must be set to `TRUE` when adding `fit criteria` and/or `bayes_R2` during model optimization.
`usesavedfuns`	A logical (default `NULL`) to indicate whether to use the already exposed and saved `Stan` functions. Depending on whether the user have exposed Stan functions within the `bsitar()` call via `expose_functions` argument in the `bsitar()`, the `usesavedfuns` is automatically set to `TRUE` (if `expose_functions = TRUE`) or `FALSE` (if `expose_functions = FALSE`). Therefore, manual setting of `usesavedfuns` as `TRUE`/`FALSE` is rarely needed. This is for internal purposes only and mainly used during the testing of the functions and therefore should not be used by users as it might lead to unreliable estimates.
`clearenvfuns`	A logical to indicate whether to clear the exposed function from the environment (`TRUE`) or not (`FALSE`). If `NULL` (default), then `clearenvfuns` is set as `TRUE` when `usesavedfuns` is `TRUE`, and `FALSE` if `usesavedfuns` is `FALSE`.
`envir`	Environment used for function evaluation. The default is `NULL` which will set `parent.frame()` as default environment. Note that since most of post processing functions are based on brms, the functions needed for evaluation should be in the `.GlobalEnv`. Therefore, it is strongly recommended to set `envir = globalenv()` (or `envir = .GlobalEnv`). This is particularly true for the derivatives such as velocity curve.
`...`	Additional arguments passed to the `brms::loo()` function. Please see `brms::loo` for details on various options available.

Details

See loo::loo_compare() for details on model comparisons. For bgmfit objects, LOO is an alias of loo. Use method brms::add_criterion() to store information criteria in the fitted model object for later usage.

Value

If only one model object is provided, then an object of class loo is returned. If multiple objects are provided, an object of class loolist.

Author(s)

Satpal Sandhu satpal.sandhu@bristol.ac.uk

Examples


# Fit Bayesian SITAR model 

# To avoid mode estimation which takes time, the Bayesian SITAR model fit to 
# the 'berkeley_exdata' has been saved as an example fit ('berkeley_exfit').
# See 'bsitar' function for details on 'berkeley_exdata' and 'berkeley_exfit'.

# Check and confirm whether model fit object 'berkeley_exfit' exists
 berkeley_exfit <- getNsObject(berkeley_exfit)

model <- berkeley_exfit


loo_validation(model, cores = 1)

[Package bsitar version 0.2.1 Index]