Abstract
In psychological science, there is an increasing concern regarding the reproducibility of scientific findings. For instance, the Replication Project: Psychology (Open Science Collaboration, 2015) found that the proportion of successful replication in psychology was 41%. This proportion was calculated based on Cumming and Maillardet’s (2006) widely employed capture procedure (CPro) and capture percentage (CPer). Despite the popularity of CPro and CPer, we believe that using them may lead to an incorrect conclusion of (a) successful replication when the population effect sizes in the original and replicated studies are different; and (b) unsuccessful replication when the population effect sizes in the original and replicated studies are identical but their sample sizes are different. Our simulation results show that the performances of CPro and CPer become biased, such that researchers can easily make a wrong conclusion of successful/unsuccessful replication. Implications of these findings are considered in the conclusion.
Introduction
In the wake of the "replication crisis," defining what constitutes a "successful" replication has become a central challenge. While $p$-values were traditionally used, their inconsistency has led researchers to seek alternatives. One popular alternative is the Capture Procedure (CPro) and its aggregate metric, Capture Percentage (CPer).
Under CPro, a replication is deemed successful if the effect size (ES) from the original study falls within the 95% confidence interval (CI) of the replication study. Theoretically, if two studies are identical, the original ES should fall within the replication's CI approximately 83.4% of the time. However, this benchmark relies on two critical assumptions: the Homogeneity of Population Effect Sizes (HORD) and the Homogeneity of Sample Sizes (HOSS). In practice, these conditions are rarely met, leading to misleading conclusions about reproducibility.
Methodology
The authors employed Monte Carlo simulations to test the performance of CPro and CPer across 45 different experimental conditions. To ensure statistical stability, 1,000,000 capture procedures were simulated for each condition using R and RStudio. The simulations varied population effect sizes ($\delta$ from 0 to 0.8), sample sizes ($n$ from 25 to 100 per group), and effect size measures (Cohen’s $d$, robust $d$, point-biserial correlation $r_{pb}$, and the probability-of-superiority $A$). The study was divided into two parts: Study 1 violated the HORD assumption to check for false successes, and Study 2 violated the HOSS assumption to check for false failures.
Results
Study 1: Violation of Population Effect Size Identity (HORD)
When the true population effect size differed between the original and replication studies, CPro failed to detect the discrepancy. For example, even with a difference of $\delta = 0.1$, the CPer remained as high as 80%. With a moderate difference of $\delta = 0.2$, the "success" rate was still 75%. This indicates that CPro is insensitive to small but meaningful differences in underlying phenomena, often signaling a "success" when the effect has actually changed.
Study 2: Violation of Sample Size Identity (HOSS)
In Study 2, the underlying effect was kept identical, but the replication study's sample size was increased. While larger samples provide more precision (narrower CIs), they paradoxically make it harder to "capture" the original ES. When the replication study was four times larger than the original ($n=100$ vs $n=25$), the capture rate plummeted from the expected 83.4% to approximately 60%. This creates a "precision paradox" where more rigorous, larger-scale replications are more likely to be labeled as failures.
These biases were consistent across all tested effect size measures, suggesting a fundamental flaw in the logic of the capture procedure itself rather than the specific mathematics of the effect size used.
Discussion and Conclusion
The "83.4% myth" is a best-case scenario that is easily undermined by real-world research variations. The authors urge applied researchers to stop using CPro/CPer as a "gold standard" for evaluating reproducibility. Instead, the following approaches are recommended:
- Equivalence Testing: Defining a range of "equivalence" and checking if the replication falls within that boundary.
- Bayesian Approaches: Calculating the actual probability of replication success rather than relying on binary logic.
- Nuanced Interpretation: Moving away from binary "success/failure" labels toward detailed explanations of why results might differ between original and replication efforts.
Summary updated