predReliability {predReliability}

A reliability function

Description

A function used to calculate the reliability of individual predictions given by your model and prediction function with methods described in the paper (Bosnic, Z., & Kononenko, I. (2008) <doi:10.1007/s10489-007-0084-9>). It also allows you to make a correlation test to estimate which reliability estimate is the most accurate for your model.

Usage

predReliability(
  data.test,
  data.train,
  types,
  formula,
  model.function,
  predict.function,
  ceval = F,
  nThread = 1,
  ...
)

Arguments

data.test

a data.frame object used as the testing data for your prediction model

data.train

a data.frame object used as the training data for your prediction model

types

a vector of reliability test types you want to perform c("bagv", "cnk", "lcv", "sa")

formula

a formula describing the model to be fitted

model.function

a function with arguments formula and data.frame implementing the predictive model to be evaluated. The function model must return an onject representing a fitted model.

predict.function

a function with arguments model object data.frame of testing instances that will be predicted based on the given model.

ceval

a flag whether a 10-fold correlation test should be made on the requested types (default set to false)

nThread

the number

...

extra arguments you wish to be passed to your model and prediction function

References

Bosnic, Z., & Kononenko, I. (2008). Comparison of approaches for estimating reliability of individual regression predictions. Data & Knowledge Engineering, 67(3), 504-516. Bosnic, Z., & Kononenko, I. (2008). Estimation of individual prediction reliability using the local sensitivity analysis. Applied intelligence, 29(3), 187-203. Bosnic, Z., & Kononenko, I. (2009). An overview of advances in reliability estimation of individual predictions in machine learning. Intelligent Data Analysis, 13(2), 385-401.

Examples

estimates <- c("bagv", "cnk", "lcv", "sa")
predReliability(mtcars[1,], mtcars[-1,], estimates, mpg~., rpart::rpart, predict)