Asymptotic Properties of False Discovery Rate Controlling Procedures Under Independence

Pierre NeuvialSPIRE 2008 · DOI

TL;DR

This paper provides a unified theoretical framework for analyzing the performance of False Discovery Rate (FDR) controlling procedures in multiple hypothesis testing. It introduces a general method to derive the asymptotic distribution of the False Discovery Proportion (FDP) for a broad class of procedures, enabling a direct comparison of their power and revealing novel connections between different adaptive testing strategies.

What Problem Does This Paper Address?

In modern data analysis, we often perform thousands of statistical tests simultaneously (e.g., in genomics or neuroimaging). A key challenge is controlling the rate of false rejections – declaring a hypothesis significant when it's actually true. The False Discovery Rate (FDR) is a popular metric for this, representing the expected proportion of false rejections among all rejected hypotheses. While procedures like the Benjamini-Hochberg (BH95) method control FDR under independence, researchers have developed more powerful "adaptive" procedures that aim to estimate the proportion of true null hypotheses to improve power. However, a rigorous theoretical understanding of how these adaptive procedures perform, especially their fluctuations around the expected FDR, and how they relate to each other, has been lacking. This paper aims to fill that gap by providing a general framework to analyze these procedures asymptotically.

Methods

The authors leverage a "stochastic process approach" to analyze the False Discovery Proportion (FDP). They model the FDP as a random process and use tools from empirical process theory, specifically Donsker's invariance principle, to derive the asymptotic distributions. A core contribution is the introduction of a "threshold function" framework that unifies many FDR controlling procedures. This framework allows them to analyze procedures based on their "regularity conditions," particularly Hadamard differentiability. They then apply this general theory to derive the asymptotic behavior of specific procedures, including the BH95, one-stage adaptive (FDR08, BR08), and two-stage adaptive (Sto02, BKY06) procedures. A key insight is interpreting some one-stage adaptive procedures as fixed points of iterative applications of two-stage adaptive procedures.

Datasets

Not discussed in the paper. The paper focuses on theoretical analysis and asymptotic properties, not empirical evaluation on specific datasets.

Key Insights and Ideas

  • Unified Framework: The paper introduces a powerful and flexible framework based on "threshold functions" that can analyze a wide range of FDR controlling procedures, unifying their theoretical treatment.
  • Asymptotic Equivalence: It reveals that seemingly different procedures can be "asymptotically equivalent," meaning they behave identically in the limit of a large number of tests.
  • Novel Connections: A striking finding is the connection between one-stage and two-stage adaptive procedures, where one-stage procedures can be viewed as fixed points of iterative applications of their two-stage counterparts. This provides a deeper understanding of their design and behavior.
  • Power Comparison: The general framework allows for direct comparison of the asymptotic power of different procedures by analyzing their attained asymptotic FDR.
  • Regularity Conditions: The paper clearly defines the "regularity conditions" on threshold functions and associated curves that are necessary for these asymptotic results to hold, providing practical guidance for developing new procedures.

Key Results

  • The paper derives the asymptotic distribution of the False Discovery Proportion (FDP) for a broad class of FDR controlling procedures under independence.
  • It establishes conditions under which procedures are asymptotically equivalent.
  • It demonstrates that one-stage adaptive procedures (like BR08 and FDR08) can be interpreted as fixed points of iterative applications of two-stage adaptive procedures (like BKY06 and Sto02, respectively).
  • The paper provides explicit asymptotic power comparisons between various procedures, showing when one is more powerful than another. For instance, FDR08 and Sto02 are generally more powerful than BH95.

Limitations and Future Work

  • Independence Assumption: The current work focuses on the assumption of independent p-values. The authors note that their formalism can be extended to other dependence settings for which Donsker's invariance principle holds, but this is left for future research.
  • Confidence Intervals: While the paper derives asymptotic distributions for the FDP, obtaining practical confidence intervals is challenging because these distributions depend on unknown quantities (like the proportion of true null hypotheses and the distribution function of p-values). The authors suggest that bootstrapping techniques could be explored for this purpose.
  • Z-score Procedures: The paper's framework is based on p-values. The authors suggest extending the framework to Z-score-based threshold procedures, which might offer different advantages.
  • Optimality: The paper raises the question of whether more powerful procedures exist beyond those studied, potentially by improving the estimation of the proportion of true null hypotheses or by considering broader families of testing procedures.

Summary updated

Abstract

Corrigendum at http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.ejs/1257431435
BibTeX
@inproceedings{neuvial2008asymptotic,
  author = {Pierre Neuvial},
  title = {{Asymptotic Properties of False Discovery Rate Controlling Procedures Under Independence}},
  booktitle = {SPIRE},
  year = {2008},
  doi = {10.1214/08-ejs20},
}
Figure