Abstract
Genome-wide association studies (GWAS) have identified hundreds of SNPs responsible for variation in human quantitative traits. However, genome-wide-significant associations often fail to replicate across independent cohorts, in apparent inconsistency with their apparent strong effects in discovery cohorts. This limited success of replication raises pervasive questions about the utility of the GWAS field. We identify all 332 studies of quantitative traits from the NHGRI-EBI GWAS Database with attempted replication. We find that the majority of studies provide insufficient data to evaluate replication rates. The remaining papers replicate significantly worse than expected ($p < 10^{-14}$), even when adjusting for regression-to-the-mean of effect size between discovery- and replication-cohorts termed the Winner’s Curse ($p < 10^{-16}$). We show this is due in part to misreporting replication cohort-size as a maximum number, rather than per-locus one. In 39 studies accurately reporting per-locus cohort-size for attempted replication of 707 loci in samples with similar ancestry, replication rate matched expectation (predicted 458, observed 457, $p = 0.94$). In contrast, ancestry differences between replication and discovery (13 studies, 385 loci) cause the most highly-powered decile of loci to replicate worse than expected, due to difference in linkage disequilibrium.
Introduction: The Illusion of a Replication Crisis
For over a decade, Genome-Wide Association Studies (GWAS) have been the primary tool for mapping the genetic architecture of human traits. Despite identifying thousands of variants, the field is frequently criticized for a perceived "replication crisis"—where findings from discovery cohorts fail to hold up in independent validation samples. This discrepancy is often blamed on the Winner’s Curse, a statistical phenomenon where the most significant results in a study are likely to have overestimated effect sizes due to random noise.
This paper investigates whether this crisis is a fundamental flaw in genetic research or a predictable outcome of statistical bias and reporting errors. By analyzing 332 quantitative trait studies, the authors demonstrate that when the Winner’s Curse is corrected and reporting is accurate, GWAS results are remarkably reliable.
Methods: De-biasing the Winners
The researchers performed a meta-analysis of replications using the NHGRI-EBI GWAS Catalog. To evaluate the studies, they applied a parametric Maximum Likelihood Estimation (MLE) model to "de-bias" reported effect sizes. They then used a Poisson binomial distribution to predict the expected number of replications for each study, comparing these predictions against the actual observed outcomes.
The Reporting and Data Gap
A major hurdle identified was not biological, but clerical. Over 58% of the surveyed papers did not provide enough summary data (effect sizes, standard errors, or p-values) to even evaluate their replication claims. Furthermore, many studies reported a "maximum sample size" for the entire replication phase rather than the actual "per-locus" sample size. This inflation of sample size leads to over-optimistic expectations of power, making a standard statistical outcome look like a failure to replicate.
Key Results: The Power of Statistical Correction
Across the 100 papers with sufficient data, the raw observed replication rate was only 48%, compared to a naive (uncorrected) expectation of 90.7%. Correcting for the Winner's Curse brought the predicted rate down to 53.8%, narrowing the gap significantly.
The most striking evidence came from a "gold standard" subset of 39 papers that matched ancestry between cohorts and reported accurate per-locus sample sizes. In this group, the statistical model predicted 458 replications and observed 457—a near-perfect match ($p = 0.94$).
The Role of Ancestry
When discovery and replication cohorts have different ancestries, replication rates drop. The study suggests this is largely due to differences in Linkage Disequilibrium (LD) patterns. The "proxy" SNPs identified in one population may not be as strongly linked to the causal variant in another, leading to lower observed effect sizes in the replication phase.
Conclusion
The perceived replication crisis in GWAS is largely explained by the Winner’s Curse and inconsistent reporting of sample sizes. When these factors are accounted for, the field's results are highly reproducible. This study underscores the need for more rigorous reporting standards and highlights that the statistical foundations of GWAS are sound, provided the "winners" are properly de-biased.
Summary updated