esb.test {metaumbrella}

Perform some tests for excess of significance

Description

The esb.test() function performs various tests to examine the presence of an excess of statistical significance in a given set of studies. These tests aims to determine whether there is an excess in the observed number of studies with statistically significant results compared to what could have been expected.

Usage

esb.test(
  x,
  input = "dataframe",
  n_cases = NULL,
  n_controls = NULL,
  measure = NULL,
  method.esb = "TESSPSST",
  true_effect = "UWLS",
  seed = NA,
  tau2 = NA
)

Arguments

x

a well-formatted dataset or an object of class “rma” or “meta”. If a well-formatted dataset is used, only one factor should be included.

input

the type of object used as input. It must be "dataframe", "rma" or "meta".

n_cases

vector with the number of cases of each included studies. Only required when x is an object of class “rma” or “meta”. This information can be indicated via the n_cases argument of the esb.test() function or directly when calling the rma() or meta() functions (see examples below).

n_controls

vector with the number of controls of each included studies. Only required when x is an object of class “rma” or “meta” This information can be indicated via the n_controls argument of the esb.test() function or directly when calling the rma() or meta() functions (see examples below).

measure

the measure of the effect: "SMD", "MD", "R", "Z", "G", "OR" or "logOR, "RR" or "logRR", "HR" or "logHR", "IRR" or "logIRR". If a an object of class “rma” or “meta” is used, the effect size should be either "SMD" or "OR". However, note that for “rma” objects, a SMD is systematically assumed to be a G (to respect the naming used in the metafor package). For “meta” objects, a SMD is assumed to be a G unless it is explicitly stated that this is not the case (i.e., using the method.smd = "Cohen" argument). The effect size measure used can be indicated via the measure argument of the esb.test() function or directly when calling the rma() or meta() functions (see examples below).

method.esb

the method used to conduct the test. It must be IT.binom, IT.chisq, PSST, TESS or TESSPSST (see details). Default is "TESSPSST".

true_effect

the best approximation of the true effect. It must be "largest", "UWLS" or a numeric value (see details). Default is "UWLS".

seed

an integer value used as an argument by the set.seed() function. Only used for measures "OR", "logOR, "RR", "logRR", "IRR" or "logIRR".

tau2

The tau2 value that should be used when using one of the PSST, TESS or TESSPSST methods (see details).

Details

The function starts by calculating whether each individual study has significant results (p < .05). Then, it estimates the statistical power of each individual study to detect an effect size equal to the best approximation of the true effect. The true_effect argument can be used to select the method that will be applied to estimate the true effect.

• If "largest" is entered, the true effect size is assumed to be equal to the effect size of the largest study included in the meta-analysis.

• If "UWLS" is entered, the true effect size is assumed to be equal to unrestricted weighted least squares weighted average.

• If a numeric value is entered, the true effect size is assumed to be equal to the value entered by the user (note that the value of ratios must be in their natural scale).

Last, this function performs a statistical test to determine whether the observed number of statistically significant studies is higher than expected given the mean statistical power. The method.esb argument can be used to select the test.

• If "IT.binom" is entered, the excess statistical significance test described by Ioannidis and Trikalinos (2007) is performed using a binomial exact test. This test explores whether the number of studies with statistically significant results is higher than what could have been expected given the mean statistical power to detect the best approximation of the true effect.

• If "IT.chisq" is entered, the excess statistical significance test described by Ioannidis and Trikalinos (2007) is performed using a chi-square test. This test explores whether the number of studies with statistically significant results is higher than what could have been expected given the mean statistical power to detect the best approximation of the true effect.

• If "TESS" is entered, the test of excess statistical significance (TESS) described by Stanley and colleagues (2021) is performed. This test assesses whether the proportion of excess statistical significance is larger than 5%. In this test, power calculations take into account between-study heterogeneity.

• If "PSST" is entered, the proportion of statistical significance test (PSST) described by Stanley and colleagues (2021) is performed. This is a test assessing whether the proportion of statistically significant studies is higher than what could have been expected given the mean statistical power. In this test, power calculations take into account between-study heterogeneity.

• If "TESSPSST" is entered, the function combines results of both "PSST" and "TESS" analyses. "TESSPSST" assumes an excess of statistical significance if at least one of "TESS" and "PSST" is statistically significant.

Value

The dataset contains the following columns:

method	method used to conduct the test.
p.value	p-value for the test statistic.
power	the power of each individual study to detect the best
	approximation of the true effect (true_effect) at an alpha of .05.
mean_power	the mean power of all individual studies to detect the best
	approximation of the true effect (true_effect) at an alpha of .05.
k	the total number of studies.
sig	whether each individual study has statistically significant results.
O	the total number of studies with statistically significant results.
E	the total expected number of studies with statistically significant results.

References

Ioannidis, JPA., Munafo, MR., Fusar-Poli, P., Nosek, BA., & David, SP. (2014). Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. Trends in Cognitive Sciences, 18, 235-241.

Examples

### load a well-formatted dataframe with a single factor
df <- df.SMD[df.SMD$factor == "Surgical", ]

### perform an excess significance bias directly on this dataframe
esb <- esb.test(df, measure = "SMD", input = "dataframe",
                method.esb = "IT.binom", true_effect = "largest")

### perform an excess significance bias using the umbrella function
esb.umbrella <- umbrella(df, method.esb = "IT.binom", true_effect = "largest")[[1]]$esb

### perform an excess significance bias on a rma object
### we convert the SMD into Hedges' g
G <- metaumbrella:::.estimate_g_from_d(df$value, df$n_cases, df$n_controls)
rma <- metafor::rma(yi = G$value, sei = G$se,
                    measure = "SMD",
                    ni = df$n_cases + df$n_controls,
                    data = df)

esb.rma <- esb.test(rma, n_cases = df$n_cases, input = "rma", method.esb = "IT.binom")

### perform an excess significance bias on a meta object
meta <- meta::metagen(TE = G$value, seTE = G$se,
                      sm = "SMD",
                      n.e = n_cases, n.c = n_controls,
                      data = df)

esb.meta <- esb.test(meta, input = "meta", method.esb = "IT.binom")

all.equal(esb$p.value, esb.umbrella$p.value, esb.rma$p.value, esb.meta$p.value)