Anytime Valid Inference and the Betting Revolution

This direction is merged into another direction: peeking problem. Its last version stays readable and citable here.

Version 1 of 126 papers27 conceptsfrozen 2026-10-04

In modern data science, the "peeking problem" is a ubiquitous challenge. Whether it is a product manager at Netflix checking A/B test results every hour or a clinical researcher monitoring a trial for early signs of harm, the urge to look at data as it arrives is natural. However, traditional frequentist statistics—the world of p-values and t-tests—was built for a "fixed-horizon" world. In that world, you must decide your sample size in advance, collect exactly that much data, and look at the results only once.

If you violate this rule by "peeking" and stopping early when results look significant, you are effectively "p-hacking." As Aaditya Ramdas, Peter Grünwald, and others have noted, this practice significantly inflates false positive rates, making spurious fluctuations look like real discoveries Game-theoretic Statistical Inference: Optional Sampling, Universal Inference, and Multiple Testing Based on E-values.

A new framework, often called Anytime-Valid Inference (AVI) or Game-Theoretic Statistics, has emerged to solve this. It provides tools that remain statistically valid no matter when you stop, how often you peek, or even if you decide to continue an experiment because the initial results were promising.

The Intuition of Evidence as Wealth

The core shift in this research direction is moving away from p-values and toward e-values. To understand why this works, imagine a betting game.

Suppose you want to test a null hypothesis (e.g., "this new feature has no effect on user retention"). In the game-theoretic view, you start with $1 and place bets against that null hypothesis as data arrives. If the null hypothesis is true, your expected wealth cannot grow indefinitely; in fact, it is a "martingale"—a sequence where your future expected wealth is just your current wealth.

An e-value is simply the amount of money you have at any given time. If you bet $1 and end up with $20, you have strong evidence against the null. The beauty of this approach is Ville’s Inequality, which states that the probability that your wealth ever exceeds $20 (under the null) is at most 1/20. This holds true regardless of when you stop or how many times you check your balance.

By framing testing as a betting game, researchers like Aaditya Ramdas and Boyan Duan have created "interactive" versions of classical tests, like the Wilcoxon rank-sum test, that allow for optional stopping and continuation Which Wilcoxon should we use? An interactive rank test and other alternatives.

From Confidence Intervals to Confidence Sequences

A standard confidence interval (CI) provides a 95% guarantee for a single, pre-specified sample size $n$. If you calculate a CI at $n=100, n=101, \dots, n=1000$, the probability that the true parameter stays inside all of those intervals is much lower than 95%.

Confidence Sequences (CS) are the "anytime-valid" version of intervals. A confidence sequence is a sequence of intervals $(C_n)_{n=1}^\infty$ that contains the true parameter at all times $n$ with high probability.

Steven R. Howard, Aaditya Ramdas, and colleagues developed non-asymptotic, nonparametric confidence sequences that work for any distribution with bounded observations Uniform, nonparametric, non-asymptotic confidence sequences. These sequences are "time-uniform," meaning they provide a safety net for researchers who need to monitor data streams continuously without inflating error rates.

Scaling to Real-World Complexity

While the early theory focused on simple means, recent work has extended AVI to the complex scenarios found in modern industry:

Why It Works: The Power of Martingales

The common thread across all these papers—from Sequential Kernelized Independence Testing to Anytime-valid t-tests—is the construction of e-processes.

An e-process is a sequence of e-values that is a "supermartingale" under the null hypothesis. Because supermartingales are mathematically constrained in how much they can grow by chance, they provide a rigorous "speed limit" for false discoveries. This allows for a "testing by betting" principle: if your e-process grows large, you have evidence; if it stays small, you don't.

This framework is more than just a statistical patch for peeking; it is a fundamental reimagining of how we measure evidence. As Peter Grünwald and Vladimir Vovk argue, this game-theoretic approach makes statistics "safe" for the messy, iterative, and continuous way that modern science and industry actually operate Editorial. Game-Theoretic Statistics and Safe Anytime-Valid Inference.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_anytime_valid_inference_and_the_betting__c8d3328d,
  title = {Anytime Valid Inference and the Betting Revolution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/anytime-valid-inference-and-the-betting-revolution/rdr_c8d3328dc06a4764ab1a5cecd549ef50},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_anytime_valid_inference_and_the_betting__c8d3328d_v1,
  title = {Anytime Valid Inference and the Betting Revolution},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/anytime-valid-inference-and-the-betting-revolution/rdr_c8d3328dc06a4764ab1a5cecd549ef50/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}