In traditional statistics, we usually assume that all data is collected before analysis begins. If you want to test 1,000 different genes for their association with a disease, you gather all the results and then apply a correction—like the Benjamini-Hochberg procedure—to ensure that your "False Discovery Rate" (FDR) stays below a certain threshold (often 5%).
But modern science rarely works this way. Research teams across the globe test hypotheses one after another in a continuous stream. As Adel Javanmard and Andrea Montanari argue in Online rules for control of false discovery rate and false discovery exceedance, we need a way to make decisions on the fly without knowing how many tests will be conducted in the future, while still maintaining rigorous statistical integrity.
The Alpha-Wealth Mechanism
The core intuition behind online testing is to treat the significance level ($\alpha$) as a finite resource or "wealth."
Imagine you start with a small budget of $\alpha$-wealth. Every time you perform a test, you "spend" a portion of that wealth. If the test is inconclusive (you fail to reject the null hypothesis), that wealth is gone. However, if you make a discovery (a rejection), you "earn" wealth back. This earned wealth can then be used to fund future tests.
As D. S. Robertson and James Wason explain in Online Multiple Hypothesis Testing, this "pay-as-you-go" system allows a stream of experiments to continue indefinitely, provided the researchers continue to find genuine effects that replenish their budget.
Why Ordering is Everything
In a batch setting, the order in which you look at your results doesn't matter. In an online setting, order is the primary determinant of power.
If you start your stream by testing 100 "long-shot" hypotheses that all turn out to be false, you will quickly exhaust your $\alpha$-wealth and be unable to prove even a very strong effect that appears at position 101. Conversely, if you prioritize hypotheses that are likely to be true, you accumulate a massive "buffer" of wealth early on.
Adel Javanmard and Andrea Montanari demonstrate in On Online Control of False Discovery Rate that by using side information to front-load likely discoveries, online procedures can actually achieve higher statistical power than traditional offline benchmarks. The early wins generate a surplus that allows for more aggressive testing of subtle effects later in the stream.
Robustness to Noisy Priors
A major risk of ordered testing is that our "prior knowledge" might be wrong. If we are too aggressive in spending wealth on what we think are winners, a few early mistakes can terminate the entire discovery process.
To solve this, Lihua Lei and William Fithian developed Adaptive SeqStep in Power of Ordered Hypothesis Testing. Instead of a fragile "stop-at-the-first-sign-of-noise" approach, their method maintains a running estimate of the false discovery rate. This makes the system persistent; even if the initial ordering is noisy or imperfect, the method remains robust and eventually reverts to standard performance levels rather than failing entirely.
Interactive and Semantic Discovery
The concept of sequential testing has recently expanded into interactive and machine-learning contexts.
In Interactive Martingale Tests for the Global Null, Aaditya Ramdas and colleagues introduced a "p-value masking" technique. This allows researchers to look at the data in a partially obscured way to decide the testing order dynamically. This "interactive" approach lets humans or algorithms prioritize the most promising leads without "double-dipping" or biasing the final results.
This logic even extends to interpreting black-box AI models. Jacopo Teneggi and Jeremias Sulam apply a sequential "betting" framework in I Bet You Did Not Mean That: Testing Semantic Importance via Betting. By treating the importance of a feature (like "wings" on a bird classifier) as a hypothesis in a game, they can rank the influence of different concepts with formal guarantees, stopping the evaluation as soon as enough "wealth" is accumulated to prove a concept's significance.