Restoring Asymptotic Normality in Adaptive Experiments

23 concepts14 papers1 proposal

Adaptive data collection algorithms—such as multi-armed bandits, reinforcement learning, and active sampling—are highly effective at maximizing rewards or minimizing sample costs. By adjusting data collection strategies in real time based on past observations, these systems quickly learn to exploit optimal actions and phase out sub-optimal ones.

However, this history-dependence introduces a severe statistical challenge. As explained by Aurélien Bibaut et al. in Demystifying Inference After Adaptive Experiments, adaptivity fundamentally violates the independent and identically distributed (i.i.d.) assumptions that underpin classical statistics. When an algorithm successfully identifies a superior option, the probability of sampling alternative actions vanishes over time. As a result, standard linear regression models and empirical estimators fail to produce normal error distributions, a phenomenon highlighted by Licong Lin et al. in Statistical Limits of Adaptive Linear Models: Low-Dimensional Estimation and Inference. Without asymptotic normality, classical frequentist tools like Z-scores and standard confidence intervals become invalid, leaving researchers unable to perform rigorous post-experiment hypothesis testing.

The Core Problem: Variance Explosion

To understand why standard estimators break down, consider Inverse Propensity Weighting (IPW). To correct for selection bias in an experiment, IPW scales each observation by the inverse of its selection probability (propensity score). In a static experiment, these propensities are fixed and safely bounded away from zero. But in an adaptive experiment, the logging policy's propensity to select a sub-optimal arm can approach zero as the algorithm learns.

When a propensity score becomes extremely small, its inverse explodes. This causes the estimator's conditional variance to fluctuate wildly and suffer from extreme inflation. As Ruohan Zhan et al. demonstrate in Policy Learning with Adaptively Collected Data, this variance explosion makes it incredibly difficult to evaluate alternative policies or construct reliable confidence intervals from data generated by aggressive exploitation.

The Solution: Variance-Stabilized Weighting

To restore statistical validity, a prominent direction in adaptive inference utilizes variance-stabilized weighting schemes. Instead of using standard IPW weights, these methods modify the estimating equations with weights explicitly designed to normalize fluctuating effective sample sizes and stabilize conditional variances across time steps.

Several distinct weighting strategies have emerged to achieve this stabilization:

Why It Works: The Martingale Central Limit Theorem

The mathematical intuition for why variance stabilization works relies on a fundamental shift from classical independent sums to martingales.

In standard statistics, the Central Limit Theorem (CLT) relies on adding independent random variables. In an adaptive setting, the error at time $t$ depends heavily on the history of choices up to time $t$. However, if we construct weights that are predictable—meaning the weight applied at time $t$ is determined entirely by information available prior to time $t$—we can preserve a crucial mathematical structure.

As shown by Mufang Ying et al. in Adaptive Linear Estimating Equations, applying predictable weights ensures that the sequence of weighted errors forms a martingale difference sequence. When you sum these weighted errors over time, the resulting cumulative sum is a martingale.

To apply the Martingale Central Limit Theorem (MCLT) and recover a predictable Gaussian distribution, two conditions must be met: the terms must form a martingale difference sequence, and their cumulative conditional variances must converge to a predictable, stable limit rather than behaving erratically. By explicitly designing the weights to stabilize these conditional variances, the data stream satisfies the conditions of the MCLT. This restores the standard asymptotic normality required to compute valid Z-scores and construct accurate confidence intervals at a fixed stopping time.

Expanding the Framework Across Domains

This core insight—using predictable, variance-stabilizing weights to unlock martingale-based inference—has been extended far beyond simple multi-armed bandit means:

By systematically controlling variance through adaptive weighting, these diverse methodologies successfully bridge the gap between optimal online data collection and rigorous offline statistical inference.

Key Foundational Work

Statistical Inference with M-Estimators on Adaptively Collected Data (2021) — contributed 4 concept elements to this direction.
Post-Contextual-Bandit Inference (2021) — contributed 2 concept elements to this direction.
Online Multi-Armed Bandits with Adaptive Inference (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_variance_stabilized_weighting_3a37cddd,
  title = {variance-stabilized weighting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variance-stabilized-weighting/rdr_3a37cddd568e405287e7aa995af6c54d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_variance_stabilized_weighting_3a37cddd_v1,
  title = {variance-stabilized weighting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variance-stabilized-weighting/rdr_3a37cddd568e405287e7aa995af6c54d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake