diagnose_design {DeclareDesign}

R Documentation

Diagnose the design

Description

Generates diagnosands from a design or simulations of a design.

Usage

diagnose_design(
  ...,
  diagnosands = NULL,
  sims = 500,
  bootstrap_sims = 100,
  make_groups = NULL,
  add_grouping_variables = NULL
)

diagnose_designs(
  ...,
  diagnosands = NULL,
  sims = 500,
  bootstrap_sims = 100,
  make_groups = NULL,
  add_grouping_variables = NULL
)

vars(...)

Arguments

...	A design or set of designs typically created using the + operator, or a data.frame of simulations, typically created by simulate_design.
diagnosands	A set of diagnosands created by declare_diagnosands. By default, these include bias, root mean-squared error, power, frequentist coverage, the mean and standard deviation of the estimate(s), the "type S" error rate (Gelman and Carlin 2014), and the mean of the inquiry(s).
sims	The number of simulations, defaulting to 500. sims may also be a vector indicating the number of simulations for each step in a design, as described for simulate_design
bootstrap_sims	Number of bootstrap replicates for the diagnosands to obtain the standard errors of the diagnosands, defaulting to 100. Set to FALSE to turn off bootstrapping.
make_groups	Add group variables within which diagnosand values will be calculated. New variables can be created or variables already in the simulations data frame selected. Type name-value pairs within the function vars, i.e. vars(significant = p.value <= 0.05).
add_grouping_variables	Deprecated. Please use make_groups instead. Variables used to generate groups of simulations for diagnosis. Added to default list: c("design", "estimand_label", "estimator", "outcome", "term")

Details

If the diagnosand function contains a group_by attribute, it will be used to split-apply-combine diagnosands rather than the intersecting column names.

If sims is named, or longer than one element, a fan-out strategy is created and used instead.

If the packages future and future.apply are installed, you can set plan to run multiple simulations in parallel.

Value

a list with a data frame of simulations, a data frame of diagnosands, a vector of diagnosand names, and if calculated, a data frame of bootstrap replicates.

Examples

design <- 
  declare_model(
    N = 500, 
    U = rnorm(N),
    Y_Z_0 = U, 
    Y_Z_1 = U + rnorm(N, mean = 2, sd = 2)
  ) + 
  declare_inquiry(ATE = mean(Y_Z_1 - Y_Z_0)) + 
  declare_assignment(Z = complete_ra(N)) +
  declare_measurement(Y = reveal_outcomes(Y ~ Z)) + 
  declare_estimator(Y ~ Z, inquiry = "ATE")

## Not run: 
# using built-in defaults:
diagnosis <- diagnose_design(design)
reshape_diagnosis(diagnosis, select = "Power")

## End(Not run)

## Not run: 
# Adding a group for within group diagnosis:
diagnosis <- diagnose_design(design, 
  make_groups = vars(significant = p.value <= 0.05),
  )
diagnosis

diagnosis <- diagnose_design(design, 
  make_groups = 
    vars(effect_size = 
      cut(estimand, quantile(estimand, (0:4)/4), 
          include.lowest = TRUE)),
  )
diagnosis

## End(Not run)

# using a user-defined diagnosand
my_diagnosand <- declare_diagnosands(absolute_error = mean(abs(estimate - estimand)))

## Not run: 
diagnosis <- diagnose_design(design, diagnosands = my_diagnosand)
diagnosis

get_diagnosands(diagnosis)

get_simulations(diagnosis)


## End(Not run)
# Using an existing data frame of simulations
## Not run: 
simulations <- simulate_design(designs, sims = 2)
diagnosis   <- diagnose_design(simulations_df = simulations_df)


## End(Not run)

# If you do not specify diagnosands, the function default_diagnosands() is used, 
#   which is reproduced below.

alpha <- 0.05

default_diagnosands <- 
  declare_diagnosands(
   mean_estimand = mean(estimand),
   mean_estimate = mean(estimate),
   bias = mean(estimate - estimand),
   sd_estimate = sqrt(pop.var(estimate)),
   rmse = sqrt(mean((estimate - estimand) ^ 2)),
   power = mean(p.value <= alpha),
   coverage = mean(estimand <= conf.high & estimand >= conf.low)
  )

# A longer list of useful diagnosands might include:

extended_diagnosands <- 
  declare_diagnosands(
    mean_estimand = mean(estimand),
    mean_estimate = mean(estimate),
    bias = mean(estimate - estimand),
    sd_estimate = sd(estimate),
    rmse = sqrt(mean((estimate - estimand) ^ 2)),
    power = mean(p.value <= alpha),
    coverage = mean(estimand <= conf.high & estimand >= conf.low),
    mean_se = mean(std.error),
    type_s_rate = mean((sign(estimate) != sign(estimand))[p.value <= alpha]),
    exaggeration_ratio = mean((estimate/estimand)[p.value <= alpha]),
    var_estimate = pop.var(estimate),
    mean_var_hat = mean(std.error^2),
    prop_pos_sig = estimate > 0 & p.value <= alpha,
    mean_ci_length = mean(conf.high - conf.low)
  )
  

---