EvaluateRule {DevTreatRules}

Evaluate a Treatment Rule

Description

Perform principled evaluation of a treatment rule (using the IPW approach to account for potential confounding) on a dataset that is independent of the development dataset on which the rule was developed, either to perform model selection (with a validation dataset) or to obtain trustworthy estimates of performance for a pre-specified treatment rule (with an evaluation dataset).

Usage

EvaluateRule(
  evaluation.data,
  BuildRule.object = NULL,
  B = NULL,
  study.design,
  name.outcome,
  type.outcome,
  desirable.outcome,
  separate.propensity.estimation = TRUE,
  clinical.threshold = 0,
  name.treatment,
  names.influencing.treatment,
  names.influencing.rule,
  propensity.method = NULL,
  show.treat.all = TRUE,
  show.treat.none = TRUE,
  truncate.propensity.score = TRUE,
  truncate.propensity.score.threshold = 0.05,
  observation.weights = NULL,
  additional.weights = rep(1, nrow(evaluation.data)),
  lambda.choice = c("min", "1se"),
  propensity.k.cv.folds = 10,
  bootstrap.CI = FALSE,
  bootstrap.CI.replications = 1000,
  bootstrap.type = "basic"
)

Arguments

evaluation.data

A data frame representing the *validation* or *evaluation* dataset used to estimate the performance of a rule that was developed on an independent development dataset.

BuildRule.object

The object returned by the BuildRule() function. Defaults to NULL but is required if a treatment rule is not provided in the B argument. Only one of BuildRule.object and B should be specified.

B

A numeric vector representing a pre-specified treatment rule, which must have length equal to the number of rows in evaluation.data and elements equal to 0/FALSE indicating no treatment and 1/TRUE indicating treatment. Defaults to NULL but is required if BuildRule.object is not specified. Only one of BuildRule.object and B should be specified.

study.design

Either ‘observational’, ‘RCT’, or ‘naive’. For the observational design, the function will use inverse-probability-of-treatment observation weights (IPW) based on estimated propensity scores with predictors names.influencing.treatment; for the RCT design, the function will use IPW based on propensity scores equal to the observed sample proportions; for the naive design, all observation weights will be uniformly equal to 1.

name.outcome

A character indicating the name of the outcome variable in evaluation.data.

type.outcome

Either ‘binary’ or ‘continuous’, the form of name.outcome.

desirable.outcome

A logical equal to TRUE if higher values of the outcome are considered desirable (e.g. for a binary outcome, 1 is more desirable than 0). The OWL.framework and OWL approaches to treatment rule estimation require a desirable outcome.

separate.propensity.estimation

A logical equal to TRUE if propensity scores should be estimated separately in the test-positives and test-negatives subpopulations and equal to FALSE if propensity scores should be estimated in the combined sample. Default is TRUE.

clinical.threshold

A numeric equal to a positive number above which the predicted outcome under treatment must be superior to the predicted outcome under control for treatment to be recommended. Only used when BuildRuleObject was specified and derived from the split-regression or direct-interactions approach. Default is 0.

name.treatment

A character indicating the name of the treatment variable in evaluation.data.

names.influencing.treatment

A character vector (or element) indicating the names of the variables in evaluation.data that are expected to influence treatment assignment in the current dataset. Required for study.design=‘observational’.

names.influencing.rule

A character vector (or element) indicating the names of the variables in evaluation.data that may influence response to treatment and are expected to be observed in future clinical settings.

propensity.method

One of‘logistic.regression’, ‘lasso’, or ‘ridge’. This is the underlying regression model used to estimate propensity scores (for study.design=‘observational’. If bootstrap.CI=TRUE, then propensity.method must be ‘logistic.regression’. Defaults to NULL.

show.treat.all

A logical variable dictating whether summaries for the naive rule that assigns treatment to all observations are reported, which help put the performance of the estimated treatment rule in context. Default is TRUE

show.treat.none

A logical variable dictating whether summaries for the naive rule that assigns treatment to no observations are reported, which help put the performance of the estimated treatment rule in context. Default is TRUE

truncate.propensity.score

A logical variable dictating whether estimated propensity scores less than truncate.propensity.score.threshold away from 0 or 1 should be truncated to be truncate.propensity.score.threshold away from 0 or 1.

truncate.propensity.score.threshold

A numeric value between 0 and 0.25.

observation.weights

A numeric vector equal to the number of rows in evaluation.data that provides observation weights to be used in place of the IPW weights estimated with propensity.method. Defaults to NULL. Only one of the propensity.method and observation.weights should be specified.

additional.weights

A numeric vector of observation weights that will be multiplied by IPW weights in the rule evaluation stage, with length equal to the number of rows in evaluation.data.. This can be used, for example, to account for a non-representative sampling design or to apply an IPW adjustment for missingness. The default is a vector of 1s.

lambda.choice

Either ‘min’ or ‘1se’, corresponding to the s argument in predict.cv.glmnet() from the glmnet package; only used when propensity.method or rule.method is ‘lasso’ or ‘ridge’. Default is ‘min’.

propensity.k.cv.folds

An integer dictating how many folds to use for K-fold cross-validation that chooses the tuning parameter when propensity.method is ‘lasso’ or ‘ridge’. Default is 10.

bootstrap.CI

Logical indicating whether the ATE/ABR estimates returned by EvaluateRule() should be accompanied by 95% confidence intervals based on the bootstrap. Default is FALSE

bootstrap.CI.replications

An integer specifying how many bootstrap replications should underlie the computed CIs. Default is 1000.

bootstrap.type

One character element specifying the type of bootstrap CI that should be computed. Currently the only supported option is bootstrap.type=‘basic’, but this may be expanded in the future.

Value

A list with the following components

• recommended.treatment: A numeric vector of 0s and 1s, with length equal to the number of rows in evaluation.data, where a 0 indicates treatment is not recommended and a 1 indicates treatment is recommended for the corresponding observation in evaluation.data.

• fit.object: A list consisting of one of the following: the propensity scores estimated in the test-positives and in the test-negatives (if separate.propensity.estimation=TRUE, study.design=‘observational’, and observation.weights=NULL); the propensity scores estimated in the combined sample (if separate.propensity.estimation=FALSE, study.design=‘observational’, and observation.weights=NULL); and simply is simply null if study.design=‘RCT’ (in which case propensity score would just be the inverse of the sample proportion receiving treatment)

• summaries: a matrix with columns reporting the following summaries of treatment rule performance: the number of observations in evaluation.data recommended to receive treatment. (n.positives); the estimated average treatment effect among those recommended to receive treatment (ATE.positives); the number of observations in evaluation.data recommended to not receive treatment (n.negatives); the estimated average treatment effect among those recommended to not receive treatment (ATE.negatives); the estimated average benefit of using the rule, with the weighted average of ATE.positives and -1 * ATE.negatives where weights are the proportions of test-positives and test-negatives (ABR). If bootstrap.CI=TRUE, then 4 additional columns are included, showing the lower bound (LB) and upper bound (UB) of the 95% CIs for ATE.positives and ATE.negatives.

Examples

set.seed(123)
example.split <- SplitData(data=obsStudyGeneExpressions,
                                     n.sets=3, split.proportions=c(0.5, 0.25, 0.25))
development.data <- example.split[example.split$partition == "development",]
validation.data <- example.split[example.split$partition == "validation",]
one.rule <- BuildRule(development.data=development.data,
                     study.design="observational",
                     prediction.approach="split.regression",
                     name.outcome="no_relapse",
                     type.outcome="binary",
                     desirable.outcome=TRUE,
                     name.treatment="intervention",
                     names.influencing.treatment=c("prognosis", "clinic", "age"),
                     names.influencing.rule=c("age", paste0("gene_", 1:10)),
                     propensity.method="logistic.regression",
                     rule.method="glm.regression")
split.validation <- EvaluateRule(evaluation.data=validation.data,
                          BuildRule.object=one.rule,
                          study.design="observational",
                          name.outcome="no_relapse",
                          type.outcome="binary",
                          desirable.outcome=TRUE,
                          name.treatment="intervention",
                          names.influencing.treatment=c("prognosis", "clinic", "age"),
                          names.influencing.rule=c("age", paste0("gene_", 1:10)),
                          propensity.method="logistic.regression",
                          bootstrap.CI=FALSE)
split.validation[c("n.positives", "n.negatives",
                       "ATE.positives", "ATE.negatives", "ABR")]