Anytime Valid Inference for Continuous Data Monitoring

240 concepts175 papers2 proposals

In classical statistics, data collection and data analysis are strictly separated by a firewall. You choose a sample size $N$ in advance, collect exactly that much data, compute a test statistic (like a $p$-value), and make a final decision.

But in modern data science, machine learning, and clinical trials, this fixed-horizon framework is a major operational bottleneck. If an A/B test shows a catastrophic drop in revenue or a clinical trial reveals a dangerous side effect, practitioners want to stop immediately. Conversely, if the results are promising but inconclusive, they want to keep collecting data.

Under standard frequentist testing, doing this invalidates the mathematical guarantees of the test. This breakdown is known as the peeking problem or optional stopping bias.

The Cost of Peeking

Why does continuous monitoring break traditional statistics? Imagine testing a null hypothesis at a nominal significance level of $\alpha = 0.05$. This means that if the null hypothesis is true, there is a 5% chance of mistakenly rejecting it. However, this guarantee only holds if you evaluate the data exactly once at the pre-determined end of the experiment.

If you "peek" at the data as it accumulates—say, after every new sample—you give the test multiple opportunities to reject the null hypothesis due to temporary statistical fluctuations. As shown by Ivana Malenica et al. et al. in their work on personalized clinical trials (Anytime-valid inference in N-of-1 trials), applying standard frequentist tests to streaming data causes a rapid accumulation of Type I errors. If you monitor an experiment indefinitely, the probability of falsely claiming a significant result approaches 100%, even if there is no real effect.

As Nikos Karampatziakis et al. et al. point out in Off-Policy Confidence Sequences, this optional stopping behavior destroys the nominal coverage of standard confidence intervals. This creates a painful tradeoff between data efficiency and mathematical validity (Matteo Zecchin et al. et al. in Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection).

To resolve this, researchers have developed anytime-valid inference: a framework providing mathematical guarantees that remain valid at any arbitrary stopping time, completely immune to the peeking problem.

From Fixed Horizons to Time-Uniform Bounds

To understand why anytime-valid inference works, we must shift our perspective from fixed-time bounds to time-uniform bounds.

A traditional confidence interval $C_N$ guarantees that for a fixed sample size $N$, the true parameter $\theta$ lies within the interval with high probability:

$$\mathbb{P}(\theta \in C_N) \ge 1 - \alpha$$

In contrast, an anytime-valid confidence sequence $(C_t)_{t=1}^\infty$ guarantees that the true parameter remains inside the interval across all time steps simultaneously:

$$\mathbb{P}(\forall t \ge 1, \theta \in C_t) \ge 1 - \alpha$$

Notice the crucial difference: the universal quantifier ($\forall t$) is inside the probability operator. This means a practitioner can look at the confidence sequence at time $t=10$, $t=100$, or $t=10,000$, and stop the experiment the moment the interval shrinks enough to exclude zero or a baseline value, without inflating the false positive rate.

The Core Mechanism: Martingales and Ville's Inequality

The mathematical engine driving anytime-valid inference is the theory of martingales. A martingale is a rigorous model of a fair game. If $M_t$ represents your wealth in a betting game at time $t$, the game is a martingale if your expected wealth at the next step, given all past outcomes, is exactly equal to your current wealth: $\mathbb{E}[M_{t+1} \mid M_1, \dots, M_t] = M_t$.

The foundational tool for anytime-valid inference is Ville’s inequality, which acts as a time-uniform version of Markov's inequality. It states that for any non-negative supermartingale (a game where your expected wealth either stays the same or decreases) $M_t$ starting at $M_0 = 1$, the probability that it ever exceeds a threshold $1/\alpha$ over an infinite horizon is bounded by $\alpha$:

$$\mathbb{P}(\exists t \ge 1: M_t \ge \frac{1}{\alpha}) \le \alpha$$

This inequality provides an elegant solution to the peeking problem. If we can construct a non-negative process that behaves like a fair game under the null hypothesis, we can monitor it continuously. If our "wealth" ever crosses the threshold $1/\alpha$, we can stop and safely reject the null hypothesis, secure in the knowledge that the probability of this happening by pure chance is at most $\alpha$.

Testing by Betting and E-Values

This martingale perspective has crystallized into a paradigm known as testing by betting, formalized through the concept of $e$-values (Wouter M Koolen et al. et al. in Safe Testing).

An $e$-value is a non-negative random variable $E$ whose expected value under the null hypothesis is at most 1 ($\mathbb{E}_0[E] \le 1$). In a sequential setting, we track an $e$-process—a sequence of $e$-values $(E_t)_{t=1}^\infty$ that forms a non-negative supermartingale under the null hypothesis.

Instead of calculating a $p$-value, a researcher allocates a virtual budget to "bet" against the null hypothesis. If the null hypothesis is false and the data aligns with the alternative hypothesis, the $e$-process grows exponentially. If the null hypothesis is true, the investigator cannot expect to get rich.

This framework has been applied to diverse domains:

Designing Tight and Robust Sequences

To make anytime-valid inference practical, the confidence sequences must shrink quickly as more data arrives. Early sequential methods relied on the Law of the Iterated Logarithm (LIL), which adds a small $\log \log t$ overhead to standard concentration bounds to guarantee time-uniform validity (Shengjia Zhao et al. et al. in Adaptive Concentration Inequalities for Sequential Decision Problems).

Modern research focuses on optimizing these boundaries for specific data structures and distributions:

Finite Populations and Quantiles

When sampling without replacement from a finite population, uncertainty should naturally collapse to zero once the entire population is observed. The authors (Aaditya Ramdas et al. et al. in Confidence sequences for sampling without replacement) designed confidence sequences that explicitly exploit this property, yielding tighter bounds than replacement-based alternatives. Similarly, Aaditya Ramdas et al. et al. developed tight confidence sequences for sequential quantile estimation (Sequential estimation of quantiles with applications to A/B testing and best-arm identification), accelerating best-arm identification algorithms.

Heavy Tails and Corruptions

Standard concentration bounds often assume well-behaved, sub-Gaussian data. When data exhibits heavy tails or infinite variance, standard averages become unstable. To address this, the authors (Aaditya Ramdas et al. et al. in Catoni-style confidence sequences for heavy-tailed mean estimation) adapted robust Catoni-style influence functions to sequential settings, achieving tight sub-Gaussian width rates even under heavy-tailed distributions. This was pushed further to infinite-variance settings by Guanhua Fang et al. et al. (Catoni-type Confidence Sequences under Infinite Variance).

For environments where data streams might be maliciously manipulated, the authors (Aaditya Ramdas et al. et al. in Huber-robust confidence sequences) introduced Huber-robust confidence sequences, ensuring that time-uniform guarantees remain valid even if a fraction of the data stream is arbitrarily corrupted.

Real-World Applications

Anytime-valid inference is particularly valuable in settings where data collection and decision-making are tightly coupled.

Multi-Armed Bandits and Off-Policy Evaluation

In multi-armed bandits, an agent balances reward maximization and pure exploration. To identify the best arm as quickly as possible, the agent needs to stop sampling an arm the moment it is statistically proven to be suboptimal. The authors (Brian M Cho et al. et al. in Reward Maximization for Pure Exploration: Minimax Optimal Good Arm Identification for Nonparametric Multi-Armed Bandits) and Ian Waudby-Smith et al. et al. (Anytime-valid off-policy inference for contextual bandits) demonstrate that confidence sequences allow for optimal arm identification and off-policy evaluation under adaptive logging policies without inflating error rates.

Modern A/B Testing

In digital experimentation, companies continuously track metrics like click-through rates and revenue. The authors (VIKAS DEEP et al. et al. in Asymptotically Optimal and Computationally Efficient Average Treatment Effect Estimation in A/B testing) show that combining adaptive participant allocation with anytime-valid confidence intervals allows platforms to safely monitor experiments in real-time, cutting the required sample size by up to 50% compared to fixed-horizon designs.

Streaming LLM Detection

With the explosion of generative AI, platforms must monitor continuous text streams to detect machine-generated content. Applying a static detection model repeatedly to a live stream causes an accumulation of false alarms over time. The authors (Jun-Kun Wang et al. et al. in Online Detecting LLM-Generated Texts via Sequential Hypothesis Testing by Betting) resolve this by framing streaming LLM detection as a sequential betting game, guaranteeing a controlled false-alarm rate over indefinite horizons.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_peeking_problem_9d565848,
  title = {peeking problem},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/peeking-problem/rdr_9d56584827b14c8380f3c467cf6eb711},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_peeking_problem_9d565848_v1,
  title = {peeking problem},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/peeking-problem/rdr_9d56584827b14c8380f3c467cf6eb711/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake