In modern data science, the "peeking problem" is a ubiquitous challenge. Whether it is a product manager at Netflix checking A/B test results every hour or a clinical researcher monitoring a trial for early signs of harm, the urge to look at data as it arrives is natural. However, traditional frequentist statistics—the world of p-values and t-tests—was built for a "fixed-horizon" world. In that world, you must decide your sample size in advance, collect exactly that much data, and look at the results only once.
If you violate this rule by "peeking" and stopping early when results look significant, you are effectively "p-hacking." As Aaditya Ramdas, Peter Grünwald, and others have noted, this practice significantly inflates false positive rates, making spurious fluctuations look like real discoveries Game-theoretic Statistical Inference: Optional Sampling, Universal Inference, and Multiple Testing Based on E-values.
A new framework, often called Anytime-Valid Inference (AVI) or Game-Theoretic Statistics, has emerged to solve this. It provides tools that remain statistically valid no matter when you stop, how often you peek, or even if you decide to continue an experiment because the initial results were promising.
The Intuition of Evidence as Wealth
The core shift in this research direction is moving away from p-values and toward e-values. To understand why this works, imagine a betting game.
Suppose you want to test a null hypothesis (e.g., "this new feature has no effect on user retention"). In the game-theoretic view, you start with $1 and place bets against that null hypothesis as data arrives. If the null hypothesis is true, your expected wealth cannot grow indefinitely; in fact, it is a "martingale"—a sequence where your future expected wealth is just your current wealth.
An e-value is simply the amount of money you have at any given time. If you bet $1 and end up with $20, you have strong evidence against the null. The beauty of this approach is Ville’s Inequality, which states that the probability that your wealth ever exceeds $20 (under the null) is at most 1/20. This holds true regardless of when you stop or how many times you check your balance.
By framing testing as a betting game, researchers like Aaditya Ramdas and Boyan Duan have created "interactive" versions of classical tests, like the Wilcoxon rank-sum test, that allow for optional stopping and continuation Which Wilcoxon should we use? An interactive rank test and other alternatives.
From Confidence Intervals to Confidence Sequences
A standard confidence interval (CI) provides a 95% guarantee for a single, pre-specified sample size $n$. If you calculate a CI at $n=100, n=101, \dots, n=1000$, the probability that the true parameter stays inside all of those intervals is much lower than 95%.
Confidence Sequences (CS) are the "anytime-valid" version of intervals. A confidence sequence is a sequence of intervals $(C_n)_{n=1}^\infty$ that contains the true parameter at all times $n$ with high probability.
Steven R. Howard, Aaditya Ramdas, and colleagues developed non-asymptotic, nonparametric confidence sequences that work for any distribution with bounded observations Uniform, nonparametric, non-asymptotic confidence sequences. These sequences are "time-uniform," meaning they provide a safety net for researchers who need to monitor data streams continuously without inflating error rates.
Scaling to Real-World Complexity
While the early theory focused on simple means, recent work has extended AVI to the complex scenarios found in modern industry:
- Causal Inference: In many A/B tests, we care about the Average Treatment Effect (ATE). Ian Waudby-Smith and Edward H. Kennedy introduced Asymptotic Confidence Sequences (AsympCS), which act as a time-uniform version of the Central Limit Theorem Time-uniform central limit theory and asymptotic confidence sequences. This allows for "doubly robust" sequential causal inference, where we can adjust for covariates while monitoring results in real-time Doubly robust confidence sequences for sequential causal inference.
- Software Deployment: At Netflix, researchers Michael Lindon and Chris Sanden use sequential testing to detect software regressions in "canary" deployments. By using anytime-valid tests of stochastic ordering, they can stop a bad deployment in seconds rather than waiting for a fixed 30-minute window Rapid Regression Detection in Software Deployments through Sequential Testing.
- Delayed Outcomes: A major hurdle in online experiments is that users don't react instantly. Michael Lindon and Nathan Kallus developed design-based AVI that remains valid even when outcomes are delayed and users join the experiment at different times Design-Based Anytime-Valid Inference for Randomized Experiments with Delayed Outcomes and Staggered Entry.
Why It Works: The Power of Martingales
The common thread across all these papers—from Sequential Kernelized Independence Testing to Anytime-valid t-tests—is the construction of e-processes.
An e-process is a sequence of e-values that is a "supermartingale" under the null hypothesis. Because supermartingales are mathematically constrained in how much they can grow by chance, they provide a rigorous "speed limit" for false discoveries. This allows for a "testing by betting" principle: if your e-process grows large, you have evidence; if it stays small, you don't.
This framework is more than just a statistical patch for peeking; it is a fundamental reimagining of how we measure evidence. As Peter Grünwald and Vladimir Vovk argue, this game-theoretic approach makes statistics "safe" for the messy, iterative, and continuous way that modern science and industry actually operate Editorial. Game-Theoretic Statistics and Safe Anytime-Valid Inference.