Adaptive Storey's null proportion estimator

Zijun GaoarXiv 2023 · DOI · arXiv

Abstract

False discovery rate (FDR) is a commonly used criterion in multiple testing and the Benjamini-Hochberg (BH) procedure is arguably the most popular approach with FDR guarantee. To improve power, the adaptive BH procedure has been proposed by incorporating various null proportion estimators, among which Storey's estimator has gained substantial popularity. The performance of Storey's estimator hinges on a critical hyper-parameter, where a pre-fixed configuration lacks power and existing data-driven hyper-parameters compromise the FDR control. In this work, we propose a novel class of adaptive hyper-parameters and establish the FDR control of the associated BH procedure using a martingale argument. Within this class of data-driven hyper-parameters, we present a specific configuration designed to maximize the number of rejections and characterize the convergence of this proposal to the optimal hyper-parameter under a commonly-used mixture model. We evaluate our adaptive Storey's null proportion estimator and the associated BH procedure on extensive simulated data and a motivating protein dataset. Our proposal exhibits significant power gains when dealing with a considerable proportion of weak non-nulls or a conservative null distribution.

Introduction

In high-throughput fields like genomics and proteomics, researchers simultaneously test thousands of hypotheses. Managing the resulting false discoveries is typically handled by the Benjamini-Hochberg (BH) procedure, which controls the False Discovery Rate (FDR). However, the original BH procedure is often conservative because it assumes a worst-case scenario where nearly all tested hypotheses are null.

To reclaim this "wasted" statistical budget, researchers use Adaptive BH procedures. These methods first estimate the proportion of null hypotheses ($\pi_0$) and then adjust the testing threshold. Storey’s estimator is a widely used tool for this purpose, but its effectiveness depends on a hyper-parameter ($\lambda$). Traditionally, choosing $\lambda$ has presented a "catch-22": manual selection often lacks power, while data-driven selection has historically lacked rigorous mathematical guarantees for FDR control.

Methodology: The Martingale Approach

The authors introduce a new class of data-driven adaptive hyper-parameters ($\hat{\lambda}$) that maintain strict FDR control. The key innovation lies in framing the parameter selection as a "stopping time" problem within a martingale framework.

The two-step Adaptive Storey (AS) procedure. Step 1 determines the adaptive hyper-parameter from the information flow starting at the FDR level q. Step 2 applies the BH procedure using the resulting null proportion estimate.

Instead of examining p-values in descending order, the authors define an increasing "filtration"—a way of representing information flow as p-values are revealed from the FDR level $q$ up to 1. By proving that $\hat{\lambda}$ can be chosen as a stopping time relative to this flow, they utilize the Optional Stopping Theorem to guarantee FDR control. This allows for an optimization rule that adaptively sets $\lambda$ to minimize the estimated null proportion and maximize discoveries.

Empirical Performance and Key Results

The proposed Adaptive Storey (AS) method was evaluated against extensive simulations and a challenging protein mass spectrometry dataset. The method excels particularly when signals are "weak" (p-values that are small but not extremely so) or when the null distribution is conservative.

Performance on the protein dataset: (a) p-value histogram showing many weak signals, (b) the null proportion estimate as a function of lambda, and (c) the resulting number of rejections. The adaptive method identifies 31 rejections at its peak.

On the motivating protein dataset, the standard BH procedure identified only 16 significant proteins. By contrast, the AS method nearly doubled the discovery rate, identifying 31 rejections while maintaining the target FDR of 0.2. This gain is attributed to the method's ability to adaptively tune $\lambda$ to a region that better separates weak signals from the null background. Furthermore, the authors proved that as sample sizes increase, their data-driven $\hat{\lambda}$ converges to the theoretically optimal value.

Conclusion

This work provides a mathematically rigorous, "plug-and-play" algorithm for practitioners. By bridging the gap between automated parameter tuning and theoretical FDR guarantees, the Adaptive Storey estimator offers a powerful tool for high-dimensional data analysis. Future research may extend these proofs to handle more complex dependency structures, such as highly correlated gene networks, and explore even more refined optimal stopping rules.

Summary updated

Abstract

False discovery rate (FDR) is a commonly used criterion in multiple testing and the Benjamini-Hochberg (BH) procedure is arguably the most popular approach with FDR guarantee. To improve power, the adaptive BH procedure has been proposed by incorporating various null proportion estimators, among which Storey's estimator has gained substantial popularity. The performance of Storey's estimator hinges on a critical hyper-parameter, where a pre-fixed configuration lacks power and existing data-driven hyper-parameters compromise the FDR control. In this work, we propose a novel class of adaptive hyper-parameters and establish the FDR control of the associated BH procedure using a martingale argument. Within this class of data-driven hyper-parameters, we present a specific configuration designed to maximize the number of rejections and characterize the convergence of this proposal to the optimal hyper-parameter under a commonly-used mixture model. We evaluate our adaptive Storey's null proportion estimator and the associated BH procedure on extensive simulated data and a motivating protein dataset. Our proposal exhibits significant power gains when dealing with a considerable proportion of weak non-nulls or a conservative null distribution.
BibTeX
@misc{gao2023adaptive,
  author = {Zijun Gao},
  title = {{Adaptive Storey's null proportion estimator}},
  year = {2023},
  eprint = {2310.06357},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.06357},
}
Figure