Optimal multiple testing under a Gaussian prior on the effect sizes

Edgar Dobriban, Kristen Fortney, Stuart K. Kim, Art B. OwenBiometrika 2015 · DOI · arXiv

Abstract: We develop a new method for large-scale frequentist multiple testing with Bayesian prior information. We find optimal p-value weights that maximize the average power of the weighted Bonferroni method. Due to the nonconvexity of the optimization problem, previous methods that account for uncertain prior information are suitable for only a small number of tests. For a Gaussian prior on the effect sizes, we give an efficient algorithm that is guaranteed to find the optimal weights nearly exactly. Our method can discover new loci in genome-wide association studies and compares favourably to competitors. An open-source implementation is available.

The Multiple Testing Burden

In fields like genomics, scientists often test hundreds of thousands of hypotheses simultaneously—for example, searching for genetic variants associated with longevity or disease. To avoid false positives, researchers typically employ the Bonferroni correction. This method requires p-values to be exceptionally small to be considered "significant." While effective at controlling errors, this approach drastically reduces the "power" of a study, often causing researchers to miss genuine discoveries because the threshold is set too high.

While prior knowledge from previous studies can help prioritize certain tests, most existing weighting methods assume prior effect sizes are known exactly. In reality, prior information is always uncertain. Previous attempts to account for this uncertainty resulted in non-convex optimization problems that were computationally impossible to solve for more than a few tests. This paper bridges that gap, allowing for uncertain priors to be incorporated across millions of tests.

Methods: Bayes Weights and Gaussian Priors

The authors introduce Bayes weights. Instead of assuming a fixed effect size ($\mu$), they model the effect size as following a Gaussian distribution: $\mu_i \sim N(\eta_i, \sigma_i^2)$, where $\eta_i$ is the prior mean and $\sigma_i$ represents the uncertainty (standard error).

Key technical contributions include:

  • Optimization Framework: The problem of maximizing expected power is formulated as a constrained optimization problem.
  • Overcoming Non-convexity: The objective function is non-convex, meaning standard solvers often fail. However, the authors identified a specific structure in the Gaussian model that allows for nearly exact solutions.
  • Efficient Algorithm: They developed an algorithm with $O(J \log J)$ complexity (where $J$ is the number of tests). This allows for the processing of 2 million tests in seconds on a standard desktop—a massive leap over previous methods limited to $J=4$.
Power gain (a) and the non-convex optimization objective (b).

Key Insights: Symmetry Breaking and Robustness

One of the most striking findings is the "Symmetry Breaking" phenomenon. If a prior effect is extremely strong, the algorithm actually takes some weight away from it to give to weaker candidates. This is because a very strong candidate is likely to be discovered even with less weight; the extra weight is better spent helping a "borderline" candidate cross the significance threshold.

Surface (a) and contour (b) plots of optimal weights $W$ across prior mean $\eta$ and uncertainty $\sigma$.

Furthermore, the method follows a robust philosophy: use Bayesian logic to incorporate prior knowledge, but remain a Frequentist regarding error control. These Bayes weights are "maximin" robust; even if prior information is slightly mis-specified (e.g., the variance is off), the method still performs well, unlike simpler "filtering" methods that can fail if their arbitrary thresholds are wrong.

Results in Genomics

The authors validated their method using several high-profile Genome-Wide Association Study (GWAS) datasets, including studies on Coronary Artery Disease (CARDIoGRAM & C4D), kidney function (CKDGen), and schizophrenia (PGC).

  • Simulated Power Gains: In sparse models, Bayes weights provided a power boost of at least 50% over unweighted methods, with some "hotspots" showing 3-4 fold improvements.
  • Real-World Discoveries: When applying the method to heart disease data, Bayes weights discovered more significant loci than standard Bonferroni or previous state-of-the-art methods like Spjotvoll weights.
  • Consistency: Across multiple data pairs, Bayes weights were the only method to consistently improve results without crashing in edge cases.

Limitations and Future Work

While powerful, the method has some limitations. GWAS summary data sometimes only provides the magnitude of an effect, not the direction, requiring a two-tailed workaround that doubles the test count. Additionally, while the Gaussian prior is flexible, some genetic architectures might follow "heavy-tailed" distributions. Future work may involve more complex models of "pleiotropy" (where one gene affects multiple traits) to further refine these priors.

Open Source Note: The authors have released R and MATLAB implementations of the "iGWAS" method, ensuring these high-powered statistical tools are accessible to the broader genetics community.

Summary updated

Abstract

We develop a new method for large-scale frequentist multiple testing with Bayesian prior information. We find optimal [Formula: see text]-value weights that maximize the average power of the weighted Bonferroni method. Due to the nonconvexity of the optimization problem, previous methods that account for uncertain prior information are suitable for only a small number of tests. For a Gaussian prior on the effect sizes, we give an efficient algorithm that is guaranteed to find the optimal weights nearly exactly. Our method can discover new loci in genome-wide association studies and compares favourably to competitors. An open-source implementation is available.
BibTeX
@inproceedings{dobriban2015optimal,
  author = {Edgar Dobriban and Kristen Fortney and Stuart K. Kim and Art B. Owen},
  title = {{Optimal multiple testing under a Gaussian prior on the effect sizes}},
  booktitle = {Biometrika},
  year = {2015},
  doi = {10.1093/biomet/asv050},
}
Figure