Anytime-Valid Inference and Testing by Betting

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2473 papers893 conceptsfrozen 2026-10-04

In classical statistics, the rules of the game are set in stone before the first data point arrives. You must pre-specify your sample size, collect exactly that much data, and only then calculate a p-value or confidence interval. If you "peek" at the results early and stop because the data looks promising, you have committed a cardinal sin of frequentist inference: you have inflated your false positive rate, a practice often called p-hacking.

However, modern data science rarely works in batches. We monitor A/B tests in real-time, track the performance of deployed machine learning models, and audit language models for toxicity as they generate text. We need a statistical framework that is "anytime-valid"—one where we can stop at any moment, for any reason, without compromising the integrity of our conclusions.

The Problem with Peeking

The reason traditional methods fail under continuous monitoring is that they are designed to control the probability of an error at a fixed time $N$. If you check the results at $t=10, 20, \dots, 100$, you are giving yourself multiple opportunities to be fooled by random noise. As Aaditya Ramdas, Steven R. Howard, and colleagues demonstrate in Uniform, nonparametric, non-asymptotic confidence sequences, standard intervals fail to control Type I error when monitored continuously. To fix this, we need Confidence Sequences (CS): a sequence of intervals that contain the true parameter with high probability at every time step simultaneously.

Hypothesis Testing as a Betting Game

The most intuitive way to understand anytime-valid inference is through the lens of "testing by betting." This framework, championed by Aaditya Ramdas and Ian Waudby-Smith in Estimating means of bounded random variables by betting, reimagines a statistical test as a game between a skeptic and the null hypothesis.

  1. The Setup: You start with $1 of wealth.
  2. The Bet: For each new data point, you place a bet against the null hypothesis. If the null is true, the game is fair—you cannot expect to increase your wealth over time.
  3. The Evidence: If the null is false, and you bet correctly, your wealth grows.
  4. The Rule: You reject the null hypothesis the moment your wealth exceeds a certain threshold (e.g., $20 for a 5% significance level).

Mathematically, this wealth process is a non-negative martingale (or supermartingale) under the null hypothesis. A fundamental result in probability known as Ville’s Inequality guarantees that the probability of a non-negative martingale ever exceeding $1/\alpha$ is at most $\alpha$. This is the "magic" of anytime-validity: because the bound holds for the entire trajectory of the wealth process, you can stop the experiment the second you become "rich" enough, or keep going forever if you don't.

From P-values to E-values

This betting framework replaces the traditional p-value with the e-value. While a p-value is the probability of seeing data at least as extreme as what was observed, an e-value is simply the wealth accumulated by the skeptic.

In Hypothesis testing with e-values, Aaditya Ramdas and Ruodu Wang argue that e-values are more robust than p-values for sequential testing and multiple testing under dependence. Unlike p-values, which are difficult to combine when tests are related, e-values can be averaged or multiplied while maintaining clear statistical guarantees. As Lasse Fischer notes in Admissible online closed testing must employ e-values, e-values are actually a theoretical necessity for optimal online multiple testing.

Real-World Applications

The flexibility of anytime-valid inference has led to a surge of applications across machine learning:

The Cost of Flexibility

There is no free lunch. To be valid at any time, confidence sequences must be wider than fixed-sample confidence intervals. Aaditya Ramdas and his co-authors point out in their 2018 work that this often involves a "doubling" of the uncertainty margin. However, as shown in Comparing Sequential Forecasters by Yo Joong Choe and Aaditya Ramdas, the ability to stop an experiment early when the signal is strong often outweighs the cost of wider bounds, leading to more efficient use of data and faster decision-making.

Papers in this version

Showing 60 of 473 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 452 papers
  2. Version 1 · 2026-10-04 · 473 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_anytime_valid_inference_c9960b23,
  title = {anytime-valid inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/anytime-valid-inference/rdr_c9960b23d18b4917a86e9227693ce44f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_anytime_valid_inference_and_testing_by_b_c9960b23_v1,
  title = {Anytime-Valid Inference and Testing by Betting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/anytime-valid-inference-and-testing-by-betting/rdr_c9960b23d18b4917a86e9227693ce44f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}