In classical statistics, the rules of the game are set in stone before the first data point arrives. You must pre-specify your sample size, collect exactly that much data, and only then calculate a p-value or confidence interval. If you "peek" at the results early and stop because the data looks promising, you have committed a cardinal sin of frequentist inference: you have inflated your false positive rate, a practice often called p-hacking.
However, modern data science rarely works in batches. We monitor A/B tests in real-time, track the performance of deployed machine learning models, and audit language models for toxicity as they generate text. We need a statistical framework that is "anytime-valid"—one where we can stop at any moment, for any reason, without compromising the integrity of our conclusions.
The Problem with Peeking
The reason traditional methods fail under continuous monitoring is that they are designed to control the probability of an error at a fixed time $N$. If you check the results at $t=10, 20, \dots, 100$, you are giving yourself multiple opportunities to be fooled by random noise. As Aaditya Ramdas, Steven R. Howard, and colleagues demonstrate in Uniform, nonparametric, non-asymptotic confidence sequences, standard intervals fail to control Type I error when monitored continuously. To fix this, we need Confidence Sequences (CS): a sequence of intervals that contain the true parameter with high probability at every time step simultaneously.
Hypothesis Testing as a Betting Game
The most intuitive way to understand anytime-valid inference is through the lens of "testing by betting." This framework, championed by Aaditya Ramdas and Ian Waudby-Smith in Estimating means of bounded random variables by betting, reimagines a statistical test as a game between a skeptic and the null hypothesis.
- The Setup: You start with $1 of wealth.
- The Bet: For each new data point, you place a bet against the null hypothesis. If the null is true, the game is fair—you cannot expect to increase your wealth over time.
- The Evidence: If the null is false, and you bet correctly, your wealth grows.
- The Rule: You reject the null hypothesis the moment your wealth exceeds a certain threshold (e.g., $20 for a 5% significance level).
Mathematically, this wealth process is a non-negative martingale (or supermartingale) under the null hypothesis. A fundamental result in probability known as Ville’s Inequality guarantees that the probability of a non-negative martingale ever exceeding $1/\alpha$ is at most $\alpha$. This is the "magic" of anytime-validity: because the bound holds for the entire trajectory of the wealth process, you can stop the experiment the second you become "rich" enough, or keep going forever if you don't.
From P-values to E-values
This betting framework replaces the traditional p-value with the e-value. While a p-value is the probability of seeing data at least as extreme as what was observed, an e-value is simply the wealth accumulated by the skeptic.
In Hypothesis testing with e-values, Aaditya Ramdas and Ruodu Wang argue that e-values are more robust than p-values for sequential testing and multiple testing under dependence. Unlike p-values, which are difficult to combine when tests are related, e-values can be averaged or multiplied while maintaining clear statistical guarantees. As Lasse Fischer notes in Admissible online closed testing must employ e-values, e-values are actually a theoretical necessity for optimal online multiple testing.
Real-World Applications
The flexibility of anytime-valid inference has led to a surge of applications across machine learning:
- Model Monitoring: Aleksandr Podkopaev and Aaditya Ramdas use confidence sequences in Tracking the risk of a deployed model and detecting harmful distribution shifts to alert engineers when a model's performance degrades in production, ensuring a strictly controlled false alarm rate.
- Fairness Auditing: In Auditing Fairness by Betting, Ben Chugg and colleagues frame fairness as a sequential game, allowing auditors to stop as soon as evidence of bias is found.
- LLM Oversight: Xuanli He and others apply these tests in An Auditing Test To Detect Behavioral Shift in Language Models to detect when an LLM's behavior (like toxicity or helpfulness) shifts after an update.
- Two-Sample Testing: Teodora Pandeva and Patrick Forré have extended this to deep learning in E-Valuating Classifier Two-Sample Tests, using neural networks to learn the best betting strategies for high-dimensional data like images.
The Cost of Flexibility
There is no free lunch. To be valid at any time, confidence sequences must be wider than fixed-sample confidence intervals. Aaditya Ramdas and his co-authors point out in their 2018 work that this often involves a "doubling" of the uncertainty margin. However, as shown in Comparing Sequential Forecasters by Yo Joong Choe and Aaditya Ramdas, the ability to stop an experiment early when the signal is strong often outweighs the cost of wider bounds, leading to more efficient use of data and faster decision-making.