Abstract
Multiple hypothesis testing, a situation when we wish to consider many hypotheses, is a core problem in statistical inference that arises in almost every scientific field. In this setting, controlling the false discovery rate (FDR), which is the expected proportion of type I error, is an important challenge for making meaningful inferences. In this paper, we consider a setting where an ordered (possibly infinite) sequence of hypotheses arrives in a stream, and for each hypothesis we observe a p-value along with a set of features specific to that hypothesis. The decision whether or not to reject the current hypothesis must be made immediately at each time step, before the next hypothesis is observed. This model provides a general way of leveraging the side (contextual) information in the data to help maximize the number of discoveries while controlling the FDR. We propose a new class of powerful online testing procedures, where the rejection thresholds are learned sequentially by incorporating contextual information and previous results. We prove that any rule in this class controls online FDR under some standard assumptions. We then focus on a subclass of these procedures, based on weighting the rejection thresholds, to derive a practical algorithm that learns a parametric weight function in an online fashion to gain more discoveries. We also theoretically prove that our proposed procedures, under some easily verifiable assumptions, would lead to an increase of statistical power over a popular online testing procedure proposed by Javanmard and Montanari (2018). Finally, we demonstrate the superior performance of our procedure, by comparing it to state-of-the-art online multiple testing procedures, on both synthetic data and real data generated from different applications.
Introduction
In modern scientific research, the simultaneous testing of thousands of hypotheses—such as identifying disease-causing gene mutations—is commonplace. A significant challenge in this "multiple testing" environment is the risk of false discoveries. For instance, applying a standard 5% significance level to 1,000 tests would likely yield 50 false positives purely by chance. Traditionally, researchers employ the False Discovery Rate (FDR) to manage these "false alarms."
While offline methods like the Benjamini-Hochberg (BH) procedure are standard, many modern applications—such as A/B testing or ongoing clinical trials—require "online" FDR control. In these scenarios, hypotheses arrive in a stream, and decisions must be made immediately. Existing online methods typically rely solely on p-values, ignoring valuable contextual information (e.g., genomic location or user demographics) that could indicate a higher likelihood of a true discovery.
Methodology: Contextual Generalized Alpha-Investing
This paper introduces Contextual Generalized Alpha-Investing (CGAI) rules. This framework builds upon the "alpha-wealth" concept, where a procedure starts with a significance budget (alpha) and earns more budget for each successful discovery.
Key Innovations
- Contextual Weighting: The significance threshold ($\alpha_t$) is decomposed into a baseline (from previous results) and a weight $\omega(X_t; \theta)$ derived from the hypothesis's features ($X_t$).
- Neural Weight Function: A deep neural network (multilayer perceptron) models the weight function to handle complex, high-dimensional context.
- Online Learning: The network is trained in a single pass using a differentiable sigmoid-based approximation of the discovery rate, allowing for real-time optimization of discoveries.
- Theoretical Guarantees: The authors prove that FDR control is maintained even when using a black-box neural network, provided the underlying rule is "monotone."
Experimental Evaluation
The proposed method, CwLORD++, was evaluated against the state-of-the-art LORD++ method across several datasets.
Airway RNA-Seq
In a genomics dataset of 33,469 genes, CwLORD++ utilized gene expression counts as context. The method achieved approximately a 10% increase in discoveries compared to the p-value-only baseline while maintaining strict FDR control.
GTEx (Genotype-Tissue Expression)
Testing 464,636 SNP-gene pairs, the authors used features like SNP-to-gene distance and evolutionary conservation scores. CwLORD++ demonstrated discovery increases ranging from 2.6% to 5.5% depending on the context used.
Diabetes Prediction
Using medical records for ~10,000 patients, CwLORD++ achieved a 51% increase in power (true discoveries) over LORD++ while keeping the FDR under the 0.20 limit.
Discussion and Limitations
The "aha moment" of this research is that while p-values are noisy, side information is often stable. By "betting" more alpha-wealth on promising contexts, researchers can significantly increase statistical power. Importantly, the FDR control remains valid regardless of the neural network's internal logic.
However, some limitations remain. The power boost depends on the context being informative; noisy context may yield no benefit. Additionally, the current proof assumes independence between p-values and features under the null hypothesis, a condition that might be challenged in complex biological systems. Future work may explore fully "anytime" updates for the weight function beyond the current batch-processing approach.
Summary updated