Abstract
Multiple hypothesis testing, a situation when we wish to consider many hypotheses, is a core problem in statistical inference that arises in almost every scientific field. In this setting, controlling the false discovery rate (FDR), which is the expected proportion of type I error, is an important challenge for making meaningful inferences. In this paper, we consider the problem of controlling FDR in an online manner. Concretely, we consider an ordered, possibly infinite, sequence of hypotheses, arriving one at each timestep, and for each hypothesis we observe a p-value along with a set of features specific to that hypothesis. The decision whether or not to reject the current hypothesis must be made immediately at each timestep, before the next hypothesis is observed. The model of multi-dimensional feature set provides a very general way of leveraging the auxiliary information in the data which helps in maximizing the number of discoveries. We propose a new class of powerful online testing procedures, where the rejections thresholds (significance levels) are learnt sequentially by incorporating contextual information and previous results. We prove that any rule in this class controls online FDR under some standard assumptions. We then focus on a subclass of these procedures, based on weighting significance levels, to derive a practical algorithm that learns a parametric weight function in an online fashion to gain more discoveries. We also theoretically prove, in a stylized setting, that our proposed procedures would lead to an increase in the achieved statistical power over a popular online testing procedure proposed by Javanmard & Montanari (2018). Finally, we demonstrate the favorable performance of our procedure, by comparing it to state-of-the-art online multiple testing procedures, on both synthetic data and real data generated from different applications.
1. Introduction
In the modern data landscape, researchers and practitioners frequently encounter the "multiple testing" problem. Whether it is a tech company running thousands of sequential A/B tests or a genomic researcher analyzing millions of genetic variants, the risk of encountering false positives (Type I errors) increases with every additional test. Traditionally, statisticians manage this risk by controlling the False Discovery Rate (FDR)—the expected proportion of false discoveries among all rejections.
However, most classical FDR methods are designed for "offline" settings where all data is available simultaneously. In many real-world applications, hypotheses arrive in a stream, and a decision to reject or fail to reject must be made immediately. This paper, "Contextual Online False Discovery Rate Control" by Shiyun Chen and Shiva Kasiviswanathan, introduces a framework that not only operates in this online environment but also leverages auxiliary "contextual" information to significantly boost the power of these tests.
2. The Online FDR Challenge
In an online setting, hypotheses $H_1, H_2, \dots$ arrive sequentially. For each $H_t$, we observe a p-value $P_t$ and must decide whether to declare a discovery before the next hypothesis is observed. Existing "alpha-investing" rules, such as the LORD (Level-based Online FDR Control) family, manage a "wealth" of significance levels that is spent on tests and earned back through discoveries.
While effective, these standard methods treat every hypothesis as an anonymous p-value. They ignore the rich metadata—or context—often associated with the data, such as the genomic location of a mutation or the specific UI element in an A/B test. This paper addresses this "context gap" to improve statistical power.
3. Contextual Generalized Alpha-Investing (CGAI)
The authors propose a new class of procedures called Contextual Generalized Alpha-Investing (CGAI). The core innovation is a significance threshold that is adjusted by contextual features ($X_t$). The threshold is determined by two components:
- The Baseline: A standard online rule (like LORD++) that manages the alpha-wealth based on historical performance.
- The Weight Function: A parametric function $\omega(X_t; \theta)$, implemented as a Multi-Layer Perceptron (MLP), that takes the context $X_t$ and outputs a weight to scale the baseline threshold.
If the context suggests a hypothesis is likely to be a true discovery, the weight increases the threshold, making it easier to reject the null hypothesis. To train this neural network online without knowing the ground truth of which hypotheses are actually true, the authors maximize an Empirical Discovery Rate (EDR) using a differentiable sigmoid approximation and gradient descent.
4. Safety and Theoretical Guarantees
A critical feature of the CGAI framework is that FDR control is guaranteed regardless of the weight function's accuracy. Even if the neural network is poorly trained or the context is irrelevant, the underlying alpha-investing rules ensure the FDR remains below the target level. The context serves strictly as a tool to increase the number of true discoveries (statistical power) without compromising the safety of the statistical inference.
5. Experimental Evaluation
The authors evaluated their approach against state-of-the-art non-contextual methods across several domains:
- Healthcare (Diabetes Prediction): Using medical records for nearly 10,000 patients, the contextual method (CwLORD++) achieved a statistical power of 0.580, compared to 0.384 for the standard LORD++ method—a 51% improvement in identifying at-risk patients.
- Genomics (GTEx): In a study of over 460,000 SNP-gene pairs, incorporating context such as genomic distance and expression levels led to 6.6% more discoveries than non-contextual methods.
- Synthetic Data: In controlled simulations, the contextual approach consistently dominated standard procedures across various signal densities and FDR targets.
6. Conclusion and Future Work
By incorporating contextual features into online testing, the authors provide a powerful method for maximizing discoveries in data streams. While the current framework assumes p-value independence, future research may address the challenges of correlated p-values, which are common in genomic data. Additionally, exploring the convergence theory of the MLP-based learning objective remains an open question. Despite these challenges, the CGAI framework represents a significant advancement in making every test count in high-dimensional, sequential data environments.
Summary updated