calc_instances {CAISEr} R Documentation

## Calculates number of instances for the comparison of multiple algorithms

### Description

Calculates either the number of instances, or the power(s) of the comparisons of multiple algorithms.

### Usage

calc_instances(
ncomparisons,
d,
ninstances = NULL,
power = NULL,
sig.level = 0.05,
alternative.side = "two.sided",
test = "t.test",
power.target = "mean"
)


### Arguments

 ncomparisons number of comparisons planned d minimally relevant effect size (MRES, expressed as a standardized effect size, i.e., "deviation from H0" / "standard deviation") ninstances the number of instances to be used in the experiment. power target power for the comparisons (see Details) sig.level desired family-wise significance level (alpha) for the experiment alternative.side type of alternative hypothesis to be performed ("two.sided" or "one.sided") test type of test to be used ("t.test", "wilcoxon" or "binomial") power.target which comparison should have the desired power? Accepts "mean", "median", or "worst.case" (this last one is equivalent to the Bonferroni correction).

### Details

The main use of this routine uses the closed formula of the t-test to calculate the number of instances required for the comparison of pairs of algorithms, given a desired power and standardized effect size of interest. Significance levels of each comparison are adjusted using Holm's step-down correction (the default). The routine also takes into account whether the desired statistical power refers to the mean power (the default), median, or worst-case (which is equivalent to designing the experiment for the more widely-known Bonferroni correction). See the reference by ⁠Campelo and Wanner⁠ for details.

### Value

a list object containing the following items:

• ninstances - number of instances

• power - the power of the comparison

• d - the effect size

• sig.level - significance level

• alternative.side - type of alternative hypothesis

• test - type of test

### Sample Sizes for Nonparametric Methods

If the parameter test is set to either Wilcoxon or Binomial, this routine approximates the number of instances using the ARE of these tests in relation to the paired t.test, using the formulas (see reference by ⁠Campelo and Takahashi⁠ for details):

n.wilcox = n.ttest / 0.86 = 1.163 * n.ttest

n.binom = n.ttest / 0.637 = 1.570 * n.ttest

### Author(s)

Felipe Campelo (fcampelo@ufmg.br, f.campelo@aston.ac.uk)

### References

• P. Mathews. Sample size calculations: Practical methods for engineers and scientists. Mathews Malnar and Bailey, 2010.

• F. Campelo, F. Takahashi: Sample size estimation for power and accuracy in the experimental comparison of algorithms. Journal of Heuristics 25(2):305-338, 2019.

• F. Campelo, E. Wanner: Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances. Submitted, Journal of Heuristics, 2019.

### Examples

# Calculate sample size for mean-case power
K      <- 10   # number of comparisons
alpha  <- 0.05 # significance level
power  <- 0.9  # desired power
d      <- 0.5  # MRES

out <- calc_instances(K, d,
power     = power,
sig.level = alpha)

# Plot power of each comparison to detect differences of magnitude d
plot(1:K, out$power, type = "b", pch = 20, las = 1, ylim = c(0, 1), xlab = "comparison", ylab = "power", xaxs = "i", xlim = c(0, 11)) grid(11, NA) points(c(0, K+1), c(power, power), type = "l", col = 2, lty = 2, lwd = .5) text(1, 0.93, sprintf("Mean power = %2.2f for N = %d", out$mean.power, out$ninstances), adj = 0) # Check sample size if planning for Wilcoxon tests: calc_instances(K, d, power = power, sig.level = alpha, test = "wilcoxon")$ninstances

# Calculate power profile for predefined sample size
N <- 45
out2 <- calc_instances(K, d, ninstances = N, sig.level = alpha)

points(1:K, out2$power, type = "b", pch = 19, col = 3) text(6, .7, sprintf("Mean power = %2.2f for N = %d", out2$mean.power, out2$ninstances), adj = 0) # Sample size for worst-case (Bonferroni) power of 0.8, using Wilcoxon out3 <- calc_instances(K, d, power = 0.9, sig.level = alpha, test = "wilcoxon", power.target = "worst.case") out3$ninstances

# For median power:
out4 <- calc_instances(K, d, power = 0.9, sig.level = alpha,
test = "wilcoxon", power.target = "median")
out4$ninstances out4$power



[Package CAISEr version 1.0.17 Index]