p_orig {Replicate}

Statistical consistency of original study with replication

Description

Given the original study's effect estimate and its variance, the estimated average true effect size in the replications, and the estimated heterogeneity in the replications, computes estimated probability that the original study would have an effect estimate at least as extreme as the observed value if the original and the replications in fact are statistically consistent. Allows for heterogeneity.

Usage

p_orig(yio, vio, yr, t2, vyr)

Arguments

yio

Effect estimate in the original study.

vio

Estimated variance of effect estimate in the original study (i.e., its squared standard error).

yr

Estimated average true effect size in the replications.

t2

Estimated heterogeneity of true effect sizes in the replications.

vyr

Estimated variance of yr (i.e., its squared standard error).

Details

yr, vyr, and t2 can be estimated through, for example, random-effects meta-analysis or a mixed model fit to the individual subject data. See Mathur & VanderWeele's (under review) Appendix for details of how to specify such models.

References

1. Mathur MB & VanderWeele TJ (under review). New statistical metrics for multisite replication projects.

Examples

# replication estimates (Fisher's z scale) and SEs
# from moral credential example in Mathur and VanderWeele
# (under review)
yir = c(0.303, 0.078, 0.113, -0.055, 0.056, 0.073,
0.263, 0.056, 0.002, -0.106, 0.09, 0.024, 0.069, 0.074,
0.107, 0.01, -0.089, -0.187, 0.265, 0.076, 0.082)

seir = c(0.111, 0.092, 0.156, 0.106, 0.105, 0.057,
0.091, 0.089, 0.081, 0.1, 0.093, 0.086, 0.076,
0.094, 0.065, 0.087, 0.108, 0.114, 0.073, 0.105, 0.04)

# meta-analyze the replications
m = metafor::rma.uni( yi = yir, vi = seir^2, measure = "ZCOR" ) 

p_orig( yio = 0.210, vio = 0.062^2, 
yr = m$b, t2 = m$se.tau2^2,  vyr = m$vb )