Anytime Valid Inference via Testing by Betting

1,071 concepts413 papers1 proposal

In classical statistics, data collection and data analysis are strictly separated. To run a valid hypothesis test or construct a confidence interval, you must commit to a fixed sample size $N$ before looking at the data.

If you violate this rule—by continuously monitoring an experiment and stopping early as soon as a result looks statistically significant—you introduce a severe mathematical bias known as the "peeking problem" or $p$-hacking. Because a standard $p$-value fluctuates randomly as data arrives, repeatedly checking it guarantees that it will eventually cross the significance threshold ($\alpha = 0.05$) by pure chance, even if the null hypothesis is completely true.

Anytime-valid inference fundamentally rewrites these rules. It provides a suite of statistical tools—including confidence sequences, always-valid $p$-values, and $e-values$—that remain valid uniformly over time. This means a researcher, an automated algorithm, or an auditing system can inspect the data at every single time step, adaptively change the sample size, or stop the experiment early for any data-dependent reason, all while maintaining strict control over the false positive rate.


Hypothesis Testing as a Fair Game

The core intuition behind anytime-valid inference is to reframe statistical testing as a game of chance. Instead of calculating the probability of observing extreme data under a static distribution, we invite a skeptic to bet against the null hypothesis.

Imagine a gambler who walks into a casino with an initial wealth of \$1. The null hypothesis ($H_0$) asserts that the casino's games are perfectly fair. The gambler is allowed to place sequential bets on incoming data points.

If the null hypothesis is true, the game is mathematically fair (or unfavorable). This means the gambler's wealth process $M_t$ over time $t$ forms a non-negative martingale (or supermartingale). By definition, the expected value of a martingale at any future point is exactly equal to its current value:

$$\mathbb{E}[M_t \mid M_{t-1}, M_{t-2}, \dots, M_0] \le M_{t-1}$$

Because the gambler cannot systematically predict a fair game, their expected wealth at any point remains \$1.

However, if the null hypothesis is false (e.g., the game is rigged in a way the gambler can exploit), the gambler can deploy a betting strategy that causes their wealth to grow exponentially.


The Mathematical Engine: Ville's Inequality

Why does this betting framing solve the peeking problem? The answer lies in a foundational result from probability theory known as Ville's Inequality.

For any non-negative supermartingale $M_t$ starting at $M_0 = 1$, Ville's inequality states that the probability that the process ever exceeds a threshold $1/\alpha$ at any point in an infinite time horizon is bounded by $\alpha$:

$$\mathbb{P}\left(\sup_{t \ge 1} M_t \ge \frac{1}{\alpha}\right) \le \alpha$$

If we set our significance level to $\alpha = 0.05$, the rejection threshold is $\$1 / 0.05 = \$20$.

This inequality provides an ironclad, anytime-valid stopping rule. If the gambler's wealth hits \$20 at time $t=14$, or $t=500$, or $t=10,000$, we can immediately stop the experiment and reject the null hypothesis. Ville's inequality guarantees that the probability of a fair game *ever* generating \$20 of wealth by chance is at most 5%, completely bypassing the need to pre-specify a sample size.

This framework, often called "testing by betting," has been widely adapted across modern machine learning. For instance, Shalev Shaer et al. leverage this mechanism to test for conditional independence in data streams, while Can Chen et al. use it to continuously audit streaming text feeds to detect LLM-generated content without inflating false alarms.


From P-Values to E-Values

To operationalize testing by betting, researchers replace traditional $p$-values with e-values.

A $p$-value is a probability-constrained measure of evidence: under the null, it is uniformly distributed between 0 and 1. An $e$-value, by contrast, is an expectation-constrained measure of evidence. It is a non-negative random variable $X$ whose expected value under the null hypothesis is at most 1:

$$\mathbb{E}_{H_0}[X] \le 1$$

As Aaditya Ramdas et al. detail, $e$-values represent the gambler's wealth multiplier from a single bet. When data arrives sequentially, multiplying successive $e$-values yields an e-process—which is precisely the non-negative martingale described above.

Property P-Value E-Value / E-Process
Mathematical Constraint Probability: $\mathbb{P}(P \le \alpha) \le \alpha$ Expectation: $\mathbb{E}[X] \le 1$
Sequential Monitoring Fails; requires fixed sample size $N$ Valid at any arbitrary stopping time
Composition Difficult to combine under dependence Multiplies naturally over time steps

While $e$-values can be slightly more conservative than $p$-values in fixed-sample settings, they are vastly more efficient for sequential data. As Matteo Zecchin et al. demonstrate in hyperparameter optimization, and Kexin Huang et al. demonstrate in automated scientific hypothesis validation, $e$-processes allow automated agents to aggressively terminate unpromising trials early, saving massive computational budgets without sacrificing statistical rigor.


Confidence Sequences and the Law of the Iterated Logarithm

Hypothesis testing tells us if an effect exists, but estimation tells us how large it is. In anytime-valid inference, fixed-sample confidence intervals are replaced by confidence sequences.

A confidence sequence is a sequence of intervals $(C_t)_{t=1}^\infty$ that contain the true target parameter $\theta$ across all time steps simultaneously with high probability:

$$\mathbb{P}(\forall t \ge 1, \theta \in C_t) \ge 1 - \alpha$$

To construct an interval that holds uniformly over time, standard concentration bounds (like Hoeffding's inequality) must be modified. If an interval shrinks at the standard rate of $1/\sqrt{t}$, the random fluctuations of a data stream will eventually breach the boundary.

To prevent this, anytime-valid confidence sequences incorporate a small mathematical penalty derived from the Law of the Iterated Logarithm (LIL). The LIL dictates that the ultimate boundary of a random walk scales not just with $\sqrt{t}$, but with $\sqrt{t \log \log t}$.

As Shengjia Zhao et al. and Akshay Balsubramani et al. show, by designing boundary functions proportional to this double-logarithmic factor, one can construct "envelopes" around the running average.

Value
  ^
  |      -------------------  Upper Confidence Bound (with log log t penalty)
  |    - - - - - - - - - - -  True Parameter (θ)
  |   *   **   *  ***  * * *  Running Sample Average
  |      -------------------  Lower Confidence Bound
  |
  +-------------------------------------------------------------> Time (t)

Transitioning from a fixed-sample interval to an anytime-valid sequence incurs only a minor logarithmic width penalty, as noted by Aaditya Ramdas et al.. In exchange for this small cost, practitioners gain total freedom. For example, Nikos Karampatziakis et al. apply these sequences to off-policy evaluation, allowing production AI systems to be safely monitored and upgraded the moment a confidence sequence proves a new model outperforms an old one.


Why It Works: A Unified Perspective

Anytime-valid inference works because it shifts the foundation of statistical evidence from absolute sample sizes to conserved probabilistic properties.

Whether using the explicit boundary adjustments of the Law of the Iterated Logarithm or the elegant wealth-accumulation tracking of $e$-values, the underlying engine remains the same: mapping a stream of dependent observations into a non-negative martingale. Because martingales are structurally constrained in how high they can fluctuate by pure chance, they transform the hazardous practice of continuous data peeking into a safe, mathematically principled asset for real-time decision making.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 452 papers
  2. Version 1 · 2026-10-04 · 473 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_anytime_valid_inference_c9960b23,
  title = {anytime-valid inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/anytime-valid-inference/rdr_c9960b23d18b4917a86e9227693ce44f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_anytime_valid_inference_c9960b23_v2,
  title = {anytime-valid inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/anytime-valid-inference/rdr_c9960b23d18b4917a86e9227693ce44f/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake