Variance-Adaptive Confidence Sequences for Anytime-Valid Inference

This direction is merged into another direction: anytime-valid inference. Its last version stays readable and citable here.

Version 1 of 127 papers40 conceptsfrozen 2026-10-04

In classical statistics, a 95% confidence interval is designed for a fixed sample size $n$. If you "peek" at the data at $n/2$ and decide to stop because the result looks significant, you have violated the mathematical assumptions of the test, leading to an inflated false-positive rate—a phenomenon often called "p-hacking."

To solve this, a growing body of research led by Aaditya Ramdas, Ian Waudby-Smith, and others has developed Confidence Sequences (CSs). These are sequences of confidence intervals that remain valid at any time $t$, allowing researchers to monitor experiments continuously and stop whenever they see a result they like, without compromising statistical integrity.

The Problem: The Price of Ignorance

The simplest way to build a confidence sequence is to use a concentration inequality like Hoeffding’s. However, Hoeffding’s inequality is "variance-blind": it only cares about the range of the data (e.g., that a variable is between 0 and 1). If your data is actually very consistent (low variance), Hoeffding’s bound remains stubbornly wide because it must protect against the worst-case scenario.

As Ian Waudby-Smith and Aaditya Ramdas argue in Time-uniform central limit theory and asymptotic confidence sequences, these non-adaptive bounds are often too wide for practical use. To make anytime-valid inference efficient, we need bounds that "pinch" when the data is well-behaved.

The Mechanism: Empirical Bernstein Bounds

The core breakthrough in this field is the development of variance-adaptive boundaries. Instead of relying on a fixed global constant, these sequences use the empirical variance—the variance observed in the data stream so far—to set the width of the interval.

In Uniform, nonparametric, non-asymptotic confidence sequences, Aaditya Ramdas, Steven R. Howard, and colleagues introduced a sequential analogue to the t-test. By incorporating the observed empirical variance into the boundary, the sequence becomes much tighter when the data is less noisy. This is formalized through Empirical Bernstein inequalities, which Aaditya Ramdas and Ben Chugg recently extended to closed-form expressions for both scalars and matrices in Closed-form empirical Bernstein confidence sequences for scalars and matrices.

Intuition: Inference as Betting

Why do these sequences work? A powerful way to understand them is through the lens of game-theoretic probability.

In Estimating means of bounded random variables by betting, Ramdas and Waudby-Smith frame testing a null hypothesis (e.g., "the mean is $0.5$") as a game. You start with \$1 and bet on the outcomes of the data. If the true mean is $0.5$, it is mathematically impossible for you to increase your wealth to infinity (it is a "fair" game, or a martingale).

If you bet optimally against a specific mean $\mu$ and your wealth grows to \$20, you can be 95% sure that $\mu$ is not the true mean. The "confidence set" at any time is simply the collection of all values of $\mu$ for which you haven't "won" too much money yet.

This betting framework naturally adapts to variance:

  1. If the data is highly volatile, your bets are cautious, and the confidence set stays wide.
  2. If the data is stable, you can bet more aggressively, and the confidence set shrinks rapidly.

Shubhanshu Shekhar and Aaditya Ramdas proved in On the near-optimality of betting confidence sets for bounded means that this betting approach isn't just a neat trick—it's actually near-optimal, matching the best possible theoretical performance.

Applications: From Bandits to Model Monitoring

This "anytime-valid" property is transformative for modern data science:

By moving from fixed-time, variance-blind bounds to anytime-valid, variance-adaptive sequences, this research direction provides a safer and more efficient foundation for sequential data analysis.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_variance_adaptive_confidence_sequences_f_34c51195,
  title = {Variance-Adaptive Confidence Sequences for Anytime-Valid Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variance-adaptive-confidence-sequences-for-anytime-valid-inference/rdr_34c51195f17a49df9c1f381a155abbe2},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_variance_adaptive_confidence_sequences_f_34c51195_v1,
  title = {Variance-Adaptive Confidence Sequences for Anytime-Valid Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variance-adaptive-confidence-sequences-for-anytime-valid-inference/rdr_34c51195f17a49df9c1f381a155abbe2/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}