get_diagnostic {hmer}

Diagnostic Tests for Emulators

Description

Given an emulator, return a diagnostic measure.

Usage

get_diagnostic(
  emulator,
  targets = NULL,
  validation = NULL,
  which_diag = "cd",
  stdev = 3,
  cleaned = NULL,
  warn = TRUE,
  kfold = NULL,
  ...
)

Arguments

emulator

An object of class Emulator

targets

If desired, the target values for the output(s) of the system

validation

If provided, the emulator is tested against the outputs of these points

which_diag

Which diagnostic measure to use (choosing from cd, ce, se above)

stdev

For ‘cd’, a measure of the allowed distance from prediction and reality

cleaned

Internal for stochastic emulators

warn

Should a warning be shown if ce is chosen and no targets provided?

kfold

Mainly internal: pre-computed k-fold diagnostic results for output

...

Any other parameters to be passed through to subfunctions.

Details

An emulator's suitability can be checked in a number of ways. This function combines all current diagnostics available in the package, returning a context-dependent data.frame containing the results.

Comparison Diagnostics (cd): Given a set of points, the emulator expectation and variance are calculated. This gives a predictive range for the input point, according to the emulator. We compare this against the actual value given by the simulator: points whose emulator prediction is further away from the simulator prediction are to be investigated. This 'distance' is given by stdev, and an emulator prediction correspondingly should not be further away from the simulator value than stdev*uncertainty.

Classification Error (ce): Given a set of targets, the emulator can determine implausibility of a point with respect to the relevant target, accepting or rejecting it as appropriate. We can define a similar ‘implausibility’ function for the simulator: the combination of the two rejection schemes gives four classifications of points. Any point where the emulator would reject the point but the simulator would not should be investigated.

Standardized Error (se): The known value at a point, combined with the emulator expectation and uncertainty, can be combined to provide a standardized error for a point. This error should not be too large, in general. but the diagnostic is more useful when looking at a collection of such measures, where systematic bias or over/underconfidence can be seen.

Which of the diagnostics is performed can be controlled by the which_diag argument. If performing classification error diagnostics, a set of targets must be provided; for all diagnostics, a validation (or holdout) set can be provided. If no such set is given, then the emulator diagnostics are performed with respect to its training points, using k-fold cross-validation.

Value

A data.frame consisting of the input points, output values, and diagnostic measures.

See Also

validation_diagnostics

Other diagnostic functions: analyze_diagnostic(), classification_diag(), comparison_diag(), individual_errors(), residual_diag(), standard_errors(), summary_diag(), validation_diagnostics()

Examples

 # Use the simple SIR model via SIREmulators
 get_diagnostic(SIREmulators$ems$nS, validation = SIRSample$validation)
 # Classification error fails without the set of targets
 get_diagnostic(SIREmulators$ems$nI, SIREmulators$targets, SIRSample$validation, 'ce')
 # No validation set: k-fold cross-validation will be used.
 get_diagnostic(SIREmulators$ems$nR, which_diag = 'se')