Abstract
In a multiple testing framework, we propose a method that identifies the interval with the highest estimated false discovery rate of P-values and rejects the corresponding null hypotheses. Unlike the Benjamini-Hochberg method, which does the same but over intervals with an endpoint at the origin, the new procedure 'scans' all intervals. In parallel with Storey et al. (2004), we show that this scan procedure provides strong control of asymptotic false discovery rate. In addition, we investigate its asymptotic false non-discovery rate, deriving conditions under which it outperforms the Benjamini-Hochberg procedure. For example, the scan procedure is superior in power-law location models.
Introduction: The Multiple Testing Problem
In modern data science, researchers often perform thousands of statistical tests simultaneously, such as identifying gene expressions or detecting anomalies in sensor data. This leads to the multiple testing problem: as the number of tests increases, the probability of observing "significant" results purely by chance also increases.
To manage this, researchers typically control the False Discovery Rate (FDR)—the expected proportion of rejected null hypotheses that are actually true. The standard framework for these outcomes is summarized in the table below.
The Benjamini-Hochberg (BH) procedure is the gold standard for FDR control. It works by finding a threshold $t$ and rejecting all P-values in the interval $[0, t]$. However, BH assumes that the strongest evidence against a null hypothesis always resides near zero. In complex scenarios where the signal is faint or the noise is structured (e.g., heavy-tailed distributions), the smallest P-values might actually stem from true null hypotheses, causing BH to miss the real signal.
The Scan Procedure
We propose a Scan Procedure that generalizes the BH method. Instead of restricting the rejection region to intervals starting at zero $[0, t]$, our method "scans" all possible intervals $[s, t]$ within the $[0, 1]$ range of P-values.
The procedure follows three main steps:
-
FDR Estimation: For any interval $[s, t]$, the FDR is estimated as: $$\widehat{\text{FDR}}(s, t) = \frac{n(t - s)}{R(s, t) \vee 1}$$ where $n$ is the total number of tests and $R(s, t)$ is the number of P-values falling into that interval.
-
Optimization: The procedure searches for the longest interval $[s, t]$ where the estimated FDR is below a pre-specified level $\alpha$.
- Rejection: All hypotheses with P-values falling inside this optimal "scanned" interval are rejected.
If the optimal $s$ is 0, the scan procedure is identical to BH. However, if $s > 0$, the procedure has identified a "pocket" of signal away from the origin, effectively "skipping over" near-zero noise that would otherwise dilute the discovery.
Key Results and Performance
The scan procedure was evaluated using simulated datasets, including Normal mixture models and Cauchy (power-law) mixture models.
Normal Models
In standard Gaussian settings, the signal typically concentrates near zero. As a result, the scan procedure often converges to the same results as the BH procedure. Simulations show that the False Discovery Proportion (FDP) and False Non-discovery Proportion (FNP) for both methods are nearly indistinguishable in these cases.
Cauchy Models and Power-Law Distributions
The scan procedure truly shines in "hard" cases involving heavy-tailed distributions like the Cauchy distribution. In these models, the null hypothesis can frequently produce very small P-values that mimic signals. By allowing the rejection region to start at $s > 0$, the scan procedure maintains power where BH struggles. The authors provide a mathematical "litmus test" ($G'(0) < G'(\delta_\diamond)$) to determine when scanning will strictly outperform thresholding.
Conclusion
The scan procedure provides a robust alternative to the Benjamini-Hochberg method, offering significant power boosts in scenarios with heavy-tailed noise while maintaining comparable performance in standard Gaussian settings. While computationally more demanding than simple thresholding, it remains manageable for modern systems and offers a more flexible approach to discovery in complex data. Future work will focus on incorporating estimates of the proportion of true null hypotheses ($\pi_0$) and extending the method to handle dependent P-values.
Summary updated