In classical statistics, the "multiple comparisons problem" is a batch problem. If you test 1,000 different genes to see if they correlate with a disease, you expect about 50 to look "significant" purely by chance at a 5% threshold. To fix this, procedures like Benjamini-Hochberg (BH) require you to collect all 1,000 $p$-values, sort them, and then apply a correction.
But modern science rarely waits for a full batch. In live A/B testing at tech companies, clinical trials with interim analyses, or global genomic databases, hypotheses arrive in a continuous stream. Decisions must be made immediately: do we reject this null hypothesis now, or move on? As Adel Javanmard and Andrea Montanari point out in Online Rules for Control of False Discovery Rate and False Discovery Exceedance, traditional offline methods fail here because they require knowledge of the future. If you use up your entire "significance budget" on the first few tests, you have nothing left for the rest of eternity.
The Alpha-Wealth Economy
The breakthrough in this field is treating statistical significance as a finite resource, often called alpha-wealth. As described by D. S. Robertson and James Wason in Online Multiple Hypothesis Testing, an online algorithm starts with an initial budget of $\alpha$ (e.g., 0.05).
- Spending: For each new hypothesis, the algorithm allocates a tiny slice of its current wealth as the significance threshold for that test.
- Losing: If the test fails to reject the null, that slice of wealth is "spent" and gone forever.
- Earning: If the test results in a rejection (a discovery), the algorithm "earns" a reward, increasing its wealth for future tests.
This "pay-as-you-go" system ensures that the False Discovery Rate (FDR)—the expected ratio of false discoveries to total discoveries—stays below $\alpha$ at every point in time, even for an infinite sequence of tests.
From LORD to SAFFRON
Early algorithms like LORD (Levels based On Recent Discoveries), introduced by Adel Javanmard and Andrea Montanari, set thresholds based on the time elapsed since the last discovery. However, these methods could be conservative.
Aaditya Ramdas and colleagues improved this with SAFFRON in SAFFRON: an Adaptive Algorithm for Online Control of the False Discovery Rate. SAFFRON is "adaptive" because it tries to identify "duds"—hypotheses that are clearly null—and avoids spending much budget on them. By estimating the proportion of true nulls in the stream, SAFFRON can allocate more wealth to promising tests, significantly increasing the power to make real discoveries.
The E-Value Revolution
While $p$-values are the standard currency of statistics, they are notoriously difficult to handle under "dependency" (when tests are correlated). In Online multiple testing with e-values, Aaditya Ramdas and Ziyu Xu propose switching to e-values.
An e-value is a random variable that is small (on average 1) under the null hypothesis but can grow very large if the null is false. Unlike $p$-values, which require complex "reshaping" penalties to maintain FDR control when tests are interconnected, e-values can be averaged or multiplied more simply. This makes algorithms like e-LOND more robust and powerful in messy, real-world data streams where hypotheses are rarely independent.
Overcoming Alpha-Death and Permanent Decisions
Two major hurdles have historically limited online testing:
- Alpha-Death: In long streams, the budget can eventually dwindle to zero, making it impossible to ever reject another hypothesis. Aaditya Ramdas and others addressed this in Online control of the false discovery rate with decaying memory by introducing "decaying memory," which prioritizes recent discoveries and prevents the system from being paralyzed by ancient failures.
- Irreversibility: Traditionally, an online decision was final. In An online generalization of the (e-)Benjamini-Hochberg procedure, Lasse Fischer and Ziyu Xu introduced Online ARC (Accept-to-Reject). This allows researchers to "upgrade" a previous acceptance to a rejection as more evidence accumulates across the stream, finally bridging the gap between the flexibility of online streams and the high power of offline batch processing.