A general convex framework for multiple testing with prior information

Edgar Dobriban

Abstract

Using prior information may improve power in frequentist multiple testing. P-value weighting is a promising methodology where each test is conducted at a different level, using critical values based on independent prior data. However, existing methods are limited, and do not allow the user to specify properties of the weights that are desired in practice, such as boundedness, or monotonicity in the strength of prior evidence. Here we develop a general framework for p-value weighting based on convex optimization. This allows flexible constraints and leads to a variety of new methods, such as bounded and monotone weights, stratified weights, and smooth weights. It also recovers several existing heuristics. Finally we focus on the promising special case of bounded monotone weights. These are appealing as they increase with the strength of prior evidence, and stable because they are within pre-specified bounds. We show they have good empirical power in the analysis of genome-wide association studies, better than any weighting method we have seen.

Introduction

In large-scale scientific studies, particularly in genomics, researchers often face the "Multiple Testing Problem." When testing millions of hypotheses simultaneously—such as the association between genetic variants and a specific disease—standard corrections like the Bonferroni method become excessively conservative. These methods treat every hypothesis as equally likely to be true, ignoring valuable prior knowledge.

Statisticians have long recognized that weighting $p$-values based on prior evidence can significantly increase discovery power. However, traditional "optimal" weighting schemes, such as Spjøtvoll weights, often suffer from instability. They may assign near-zero weights to strong signals or exhibit non-monotonic behavior, where a larger expected effect size does not necessarily result in a larger weight. This paper introduces the Princessp framework to address these practical shortcomings by framing $p$-value weighting as a structured convex optimization problem.

The Princessp Framework

The core innovation of this work is the formulation of $p$-value weighting through constrained convex optimization. This approach provides a rigorous yet flexible way to incorporate "practitioner's guardrails" into the weighting process.

Convexity via Power Curves

The framework leverages the observation that for many common statistical distributions (e.g., the Gaussian "means" model), the power function—or the Receiver Operating Characteristic (ROC) curve—is concave. By exploiting this concavity, the search for optimal weights can be solved efficiently using convex programming. This mathematical foundation ensures that the resulting weights are not just heuristic but are grounded in optimization theory.

Flexible Constraints

Unlike previous "black-box" schemes, Princessp allows users to enforce specific properties:

  • Monotonicity: Hypotheses with stronger prior evidence (larger expected effect sizes) are guaranteed to receive larger weights.
  • Boundedness: Users can set lower and upper bounds on weights. This ensures that no hypothesis is entirely ignored (preventing the "killing" of unexpected signals) and that no single test dominates the overall results.

Computational Scalability

To accommodate modern datasets with tens of millions of hypotheses, the framework employs an efficient interior-point method. A key technical contribution is a subsampling algorithm used to solve the underlying KKT systems, which prevents numerical instability at massive scales and allows millions of hypotheses to be processed in seconds.

Empirical Evaluation and Applications

The Princessp framework was evaluated across several high-profile Genome-Wide Association Study (GWAS) datasets, including CARDIoGRAM (coronary artery disease), blood lipids, and schizophrenia data from the Psychiatric Genomics Consortium. These studies typically involve between 500,000 and 2.5 million single nucleotide polymorphisms (SNPs).

The use of prior information is a well-established strategy in the field, as shown in the table below:

Examples of GWAS studies utilizing prior information.

By applying bounded monotone weights to these types of problems, the framework provides a stable and powerful alternative to existing methods.

Key Results and Discussion

In head-to-head comparisons across seven genomic data settings, Bounded Monotone Weighting consistently outperformed unweighted Bonferroni corrections and other state-of-the-art methods like Bayes weighting.

A critical finding was the framework's robustness: Princessp was the only method that never decreased the number of discoveries compared to the unweighted baseline. Other "optimal" methods occasionally lost signals due to overly aggressive weighting. Furthermore, the computational efficiency of the solver makes it highly practical for real-world bioinformatics pipelines.

Limitations

While powerful, the method currently relies on "plug-in" estimates for prior effect sizes. Future work could explore a fully integrated Bayesian approach to account for the uncertainty in these priors. Additionally, while lower bounds between 0.01 and 0.5 have proven effective, the selection of these hyperparameters remains somewhat heuristic.

Conclusion

The Princessp framework bridges the gap between theoretical optimality and practical utility in multiple testing. By transforming $p$-value weighting into a convex optimization problem with user-defined constraints, it provides a robust, scalable, and powerful tool for discovery in the era of Big Data.

Summary updated

Abstract

Using prior information may improve power in frequentist multiple testing. P-value weighting is a promising methodology where each test is conducted at a different level, using critical values based on independent prior data. However, existing methods are limited, and do not allow the user to specify properties of the weights that are desired in practice, such as boundedness, or monotonicity in the strength of prior evidence. Here we develop a general framework for p-value weighting based on convex optimization. This allows flexible constraints and leads to a variety of new methods, such as bounded and monotone weights, stratified weights, and smooth weights. It also recovers several existing heuristics. Finally we focus on the promising special case of bounded monotone weights. These are appealing as they increase with the strength of prior evidence, and stable because they are within pre-specified bounds. We show they have good empirical power in the analysis of genome-wide association studies, better than any weighting method we have seen.
BibTeX
@inproceedings{dobriban2018general,
  author = {Edgar Dobriban},
  title = {{A general convex framework for multiple testing with prior information}},
  booktitle = {PubMed},
  year = {2018},
  doi = {10.1093/imaiai/iax013},
}
Figure