Optimal False Discovery Rate Control for Large Scale Multiple Testing with Auxiliary Information

Hongyuan Cao, Jun Chen, Xianyang ZhangarXiv 2021 · DOI · arXiv

Introduction

Large-scale multiple testing is a cornerstone of modern high-dimensional statistical inference. In fields such as genomics, neuroscience, and economics, researchers frequently test thousands or millions of hypotheses simultaneously. The traditional approach to managing the resulting false positives is the Benjamini-Hochberg (BH) procedure, which controls the False Discovery Rate (FDR). However, a significant limitation of the BH procedure and its variants is the assumption of "exchangeability"—the idea that every hypothesis is equally likely to be a discovery before the data is observed.

In practice, researchers often possess auxiliary information that suggests certain hypotheses are more promising than others. This information might come from previous studies, physical properties, or spatial relationships. While several methods have been developed to incorporate this data, they often require arbitrary "binning" or complex hyperparameter tuning. This paper introduces OrderShapeEM, a robust framework that adaptively incorporates auxiliary information to boost statistical power while maintaining strict FDR control.

The OrderShapeEM Framework

The proposed method is built upon a two-group mixture model. In this model, each test result (p-value) is assumed to originate from either a "null" distribution (representing no effect) or an "alternative" distribution (representing a real discovery). The key innovation lies in how the prior probability of a hypothesis being null ($\pi_{0i}$) is handled.

1. Varying Priors and Shape Constraints

Unlike standard models that assume a constant prior across all tests, OrderShapeEM allows $\pi_{0i}$ to vary for every individual hypothesis based on auxiliary information. To prevent overfitting and ensure biological plausibility, the authors impose a shape constraint. Specifically, they assume a monotone relationship: as the auxiliary relevance score increases, the probability of the hypothesis being null should decrease.

2. The OrderShapeEM Algorithm

To estimate these varying priors and the distribution of signals simultaneously, the authors developed a robust Expectation-Maximization (EM) algorithm. This algorithm integrates the Pool-Adjacent-Violators Algorithm (PAVA), a classic tool for isotonic regression. This approach makes the method "tuning-free," allowing it to scale efficiently to the massive datasets common in modern genomics without requiring manual intervention.

3. Optimal Rejection Rule

The framework employs a "step-up" procedure based on the Local False Discovery Rate (Lfdr). This rule is designed to maximize the expected number of true positives for a fixed FDR level, providing a theoretical guarantee of optimality under the model's assumptions.

Experimental Validation

Simulation Studies

The researchers conducted extensive simulations involving 10,000 hypotheses with varying signal densities (5% to 20%). The results demonstrated that OrderShapeEM consistently outperformed state-of-the-art competitors, including BH, Storey’s ST, AdaPT, and SABHA. The advantage was most pronounced when the signal was relatively weak but the auxiliary information was moderately informative.

Genome-Wide Association Studies (GWAS)

The methodology was applied to two large-scale datasets focused on Coronary Artery Disease (CAD):

  • CARDIoGRAM: A meta-analysis of 14 studies (22,000 cases).
  • C4D: A meta-analysis of 5 studies (15,000 cases).

By using one dataset as auxiliary information to inform the analysis of the other, OrderShapeEM identified 40 additional significant SNPs (genetic variants) that standard methods failed to detect. Crucially, many of these new discoveries were located in genes already known to be associated with cardiovascular and metabolic disorders, providing biological validation for the method's increased power.

Discussion and Key Insights

OrderShapeEM offers several advantages over existing "ordered" testing methods:

  • Flexibility: While some methods stop rejecting hypotheses once they hit a certain point in a ranked list, OrderShapeEM allows for "out-of-order" rejections. A hypothesis with a very strong p-value can still be rejected even if its auxiliary rank was low.
  • Robustness: By using isotonic regression (PAVA) rather than a rigid parametric formula, the model adapts to various types of auxiliary data without the risk of over-fitting.
  • Scalability: The algorithm is computationally efficient and requires no hyperparameter tuning, making it suitable for "Big Data" applications.

Limitations and Future Work

The authors note that the method's performance is tied to the quality of the auxiliary information; if the information is purely noise, the method may be slightly less powerful than the standard BH procedure. Additionally, the current model assumes that null p-values follow a uniform distribution, which may not hold in cases of composite nulls or skewed data. Future research will explore extending this framework to handle more complex structures, such as spatial dependencies or hierarchical tree structures in microbiome data.

Summary updated

Abstract

Large-scale multiple testing is a fundamental problem in high dimensional statistical inference. It is increasingly common that various types of auxiliary information, reflecting the structural relationship among the hypotheses, are available. Exploiting such auxiliary information can boost statistical power. To this end, we propose a framework based on a two-group mixture model with varying probabilities of being null for different hypotheses a priori, where a shape-constrained relationship is imposed between the auxiliary information and the prior probabilities of being null. An optimal rejection rule is designed to maximize the expected number of true positives when average false discovery rate is controlled. Focusing on the ordered structure, we develop a robust EM algorithm to estimate the prior probabilities of being null and the distribution of $p$-values under the alternative hypothesis simultaneously. We show that the proposed method has better power than state-of-the-art competitors while controlling the false discovery rate, both empirically and theoretically. Extensive simulations demonstrate the advantage of the proposed method. Datasets from genome-wide association studies are used to illustrate the new methodology.
BibTeX
@misc{cao2021optimal,
  author = {Hongyuan Cao and Jun Chen and Xianyang Zhang},
  title = {{Optimal False Discovery Rate Control for Large Scale Multiple Testing with Auxiliary Information}},
  year = {2021},
  eprint = {2103.15311},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.15311},
}
Figure