check_performance {adaptr}  R Documentation 
Calculates performance metrics for a trial specification based on
simulation results from the run_trials()
function, with bootstrapped
uncertainty measures if requested. Uses extract_results()
, which may be
used directly to extract key trial results without summarising. This function
is also used by summary()
to calculate the performance metrics presented by
that function.
check_performance(
object,
select_strategy = "control if available",
select_last_arm = FALSE,
select_preferences = NULL,
te_comp = NULL,
raw_ests = FALSE,
final_ests = NULL,
restrict = NULL,
uncertainty = FALSE,
n_boot = 5000,
ci_width = 0.95,
boot_seed = NULL,
cores = NULL
)
object 

select_strategy 
single character string. If a trial was not stopped
due to superiority (or had only 1 arm remaining, if

select_last_arm 
single logical, defaults to 
select_preferences 
character vector specifying a number of arms used
for selection if one of the 
te_comp 
character string, treatmenteffect comparator. Can be either

raw_ests 
single logical. If 
final_ests 
single logical. If 
restrict 
single character string or 
uncertainty 
single logical; if 
n_boot 
single integer (default 
ci_width 
single numeric 
boot_seed 
single integer, 
cores 

The ideal design percentage (IDP) returned is based on Viele et al, 2020 doi:10.1177/1740774519877836 (and also described in Granholm et al, 2022 doi:10.1016/j.jclinepi.2022.11.002, which also describes the other performance measures) and has been adapted to work for trials with both desirable/undesirable outcomes and nonbinary outcomes. Briefly, the expected outcome is calculated as the sum of the true outcomes in each arm multiplied by the corresponding selection probabilities (ignoring simulations with no selected arm). The IDP is then calculated as:
For desirable outcomes (highest_is_best
is TRUE
):
100 * (expected outcome  lowest true outcome) / (highest true outcome  lowest true outcome)
For undesirable outcomes (highest_is_best
is FALSE
):
100  IDP calculated for desirable outcomes
A tidy data.frame
with added class trial_performance
(to control
the number of digits printed, see print()
), with the columns
"metric"
(described below), "est"
(estimate of each metric), and the
following four columns if uncertainty = TRUE
: "err_sd"
(bootstrapped
SDs), "err_mad"
(bootstrapped MADSDs, as described in setup_trial()
and stats::mad()
), "lo_ci"
, and "hi_ci"
, the latter two corresponding
to the lower/upper limits of the percentilebased bootstrapped confidence
intervals. Bootstrap estimates are not calculated for the mininum
(_p0
) and maximum values (_p100
) of size
, sum_ys
, and ratio_ys
,
as nonparametric bootstrapping for mininum/maximum values is not
sensible  bootstrap estimates for these values will be NA
.
The following performance metrics are calculated:
n_summarised
: the number of simulations summarised.
size_mean
, size_sd
, size_median
, size_p25
, size_p75
,
size_p0
, size_p100
: the mean, standard deviation, median as well as
25, 75, 0 (min), and 100 (max) percentiles of the sample sizes
(number of patients randomised in each simulated trial) of the summarised
trial simulations.
sum_ys_mean
, sum_ys_sd
, sum_ys_median
, sum_ys_p25
,
sum_ys_p75
, sum_ys_p0
, sum_ys_p100
: the mean, standard deviation,
median as well as 25, 75, 0 (min), and 100 (max) percentiles of the
total sum_ys
across all arms in the summarised trial simulations (e.g.,
the total number of events in trials with a binary outcome, or the sums
of continuous values for all patients across all arms in trials with a
continuous outcome). Always uses all outcomes from all randomised
patients regardless of whether or not all patients had outcome data
available at the time of trial stopping (corresponding to sum_ys_all
in
results from run_trial()
).
ratio_ys_mean
, ratio_ys_sd
, ratio_ys_median
, ratio_ys_p25
,
ratio_ys_p75
, ratio_ys_p0
, ratio_ys_p100
: the mean, standard
deviation, median as well as 25, 75, 0 (min), and 100 (max)
percentiles of the final ratio_ys
(sum_ys
as described above divided
by the total number of patients randomised) across all arms in the
summarised trial simulations.
prob_conclusive
: the proportion (0
to 1
) of conclusive trial
simulations, i.e., simulations not stopped at the maximum sample size
without a superiority, equivalence or futility decision.
prob_superior
, prob_equivalence
, prob_futility
, prob_max
: the
proportion (0
to 1
) of trial simulations stopped for superiority,
equivalence, futility or inconclusive at the maximum allowed sample size,
respectively.
Note: Some metrics may not make sense if summarised simulation
results are restricted
.
prob_select_*
: the selection probabilities for each arm and for no
selection, according to the specified selection strategy. Contains one
element per arm
, named prob_select_arm_<arm name>
and
prob_select_none
for the probability of selecting no arm.
rmse
, rmse_te
: the root mean squared error of the estimates for
the selected arm and for the treatment effect, as described in
extract_results()
.
idp
: the ideal design percentage (IDP; 0100%), see Details.
extract_results()
, summary()
, plot_convergence()
,
plot_metrics_ecdf()
, check_remaining_arms()
.
# Setup a trial specification
binom_trial < setup_trial_binom(arms = c("A", "B", "C", "D"),
control = "A",
true_ys = c(0.20, 0.18, 0.22, 0.24),
data_looks = 1:20 * 100)
# Run 10 simulations with a specified random base seed
res < run_trials(binom_trial, n_rep = 10, base_seed = 12345)
# Check performance measures, without assuming that any arm is selected in
# the inconclusive simulations, with bootstrapped uncertainty measures
# (unstable in this example due to the very low number of simulations
# summarised):
check_performance(res, select_strategy = "none", uncertainty = TRUE,
n_boot = 1000, boot_seed = "base")