check_performance {adaptr} | R Documentation |
Check performance metrics for trial simulations
Description
Calculates performance metrics for a trial specification based on
simulation results from the run_trials()
function, with bootstrapped
uncertainty measures if requested. Uses extract_results()
, which may be
used directly to extract key trial results without summarising. This function
is also used by summary()
to calculate the performance metrics presented by
that function.
Usage
check_performance(
object,
select_strategy = "control if available",
select_last_arm = FALSE,
select_preferences = NULL,
te_comp = NULL,
raw_ests = FALSE,
final_ests = NULL,
restrict = NULL,
uncertainty = FALSE,
n_boot = 5000,
ci_width = 0.95,
boot_seed = NULL,
cores = NULL
)
Arguments
object |
|
select_strategy |
single character string. If a trial was not stopped
due to superiority (or had only 1 arm remaining, if
|
select_last_arm |
single logical, defaults to |
select_preferences |
character vector specifying a number of arms used
for selection if one of the |
te_comp |
character string, treatment-effect comparator. Can be either
|
raw_ests |
single logical. If |
final_ests |
single logical. If |
restrict |
single character string or |
uncertainty |
single logical; if |
n_boot |
single integer (default |
ci_width |
single numeric |
boot_seed |
single integer, |
cores |
|
Details
The ideal design percentage (IDP) returned is based on Viele et al, 2020 doi:10.1177/1740774519877836 (and also described in Granholm et al, 2022 doi:10.1016/j.jclinepi.2022.11.002, which also describes the other performance measures) and has been adapted to work for trials with both desirable/undesirable outcomes and non-binary outcomes. Briefly, the expected outcome is calculated as the sum of the true outcomes in each arm multiplied by the corresponding selection probabilities (ignoring simulations with no selected arm). The IDP is then calculated as:
For desirable outcomes (
highest_is_best
isTRUE
):
100 * (expected outcome - lowest true outcome) / (highest true outcome - lowest true outcome)
For undesirable outcomes (
highest_is_best
isFALSE
):
100 - IDP calculated for desirable outcomes
Value
A tidy data.frame
with added class trial_performance
(to control
the number of digits printed, see print()
), with the columns
"metric"
(described below), "est"
(estimate of each metric), and the
following four columns if uncertainty = TRUE
: "err_sd"
(bootstrapped
SDs), "err_mad"
(bootstrapped MAD-SDs, as described in setup_trial()
and stats::mad()
), "lo_ci"
, and "hi_ci"
, the latter two corresponding
to the lower/upper limits of the percentile-based bootstrapped confidence
intervals. Bootstrap estimates are not calculated for the minimum
(_p0
) and maximum values (_p100
) of size
, sum_ys
, and ratio_ys
,
as non-parametric bootstrapping for minimum/maximum values is not
sensible - bootstrap estimates for these values will be NA
.
The following performance metrics are calculated:
-
n_summarised
: the number of simulations summarised. -
size_mean
,size_sd
,size_median
,size_p25
,size_p75
,size_p0
,size_p100
: the mean, standard deviation, median as well as 25-, 75-, 0- (min), and 100- (max) percentiles of the sample sizes (number of patients randomised in each simulated trial) of the summarised trial simulations. -
sum_ys_mean
,sum_ys_sd
,sum_ys_median
,sum_ys_p25
,sum_ys_p75
,sum_ys_p0
,sum_ys_p100
: the mean, standard deviation, median as well as 25-, 75-, 0- (min), and 100- (max) percentiles of the totalsum_ys
across all arms in the summarised trial simulations (e.g., the total number of events in trials with a binary outcome, or the sums of continuous values for all patients across all arms in trials with a continuous outcome). Always uses all outcomes from all randomised patients regardless of whether or not all patients had outcome data available at the time of trial stopping (corresponding tosum_ys_all
in results fromrun_trial()
). -
ratio_ys_mean
,ratio_ys_sd
,ratio_ys_median
,ratio_ys_p25
,ratio_ys_p75
,ratio_ys_p0
,ratio_ys_p100
: the mean, standard deviation, median as well as 25-, 75-, 0- (min), and 100- (max) percentiles of the finalratio_ys
(sum_ys
as described above divided by the total number of patients randomised) across all arms in the summarised trial simulations. -
prob_conclusive
: the proportion (0
to1
) of conclusive trial simulations, i.e., simulations not stopped at the maximum sample size without a superiority, equivalence or futility decision. -
prob_superior
,prob_equivalence
,prob_futility
,prob_max
: the proportion (0
to1
) of trial simulations stopped for superiority, equivalence, futility or inconclusive at the maximum allowed sample size, respectively.
Note: Some metrics may not make sense if summarised simulation results arerestricted
. -
prob_select_*
: the selection probabilities for each arm and for no selection, according to the specified selection strategy. Contains one element perarm
, namedprob_select_arm_<arm name>
andprob_select_none
for the probability of selecting no arm. -
rmse
,rmse_te
: the root mean squared errors of the estimates for the selected arm and for the treatment effect, as described inextract_results()
. -
mae
,mae_te
: the median absolute errors of the estimates for the selected arm and for the treatment effect, as described inextract_results()
. -
idp
: the ideal design percentage (IDP; 0-100%), see Details.
See Also
extract_results()
, summary()
, plot_convergence()
,
plot_metrics_ecdf()
, check_remaining_arms()
.
Examples
# Setup a trial specification
binom_trial <- setup_trial_binom(arms = c("A", "B", "C", "D"),
control = "A",
true_ys = c(0.20, 0.18, 0.22, 0.24),
data_looks = 1:20 * 100)
# Run 10 simulations with a specified random base seed
res <- run_trials(binom_trial, n_rep = 10, base_seed = 12345)
# Check performance measures, without assuming that any arm is selected in
# the inconclusive simulations, with bootstrapped uncertainty measures
# (unstable in this example due to the very low number of simulations
# summarised):
check_performance(res, select_strategy = "none", uncertainty = TRUE,
n_boot = 1000, boot_seed = "base")