When running an experiment, we usually want two things: to find out which treatment works best, and to avoid harming our participants (or losing money) while we figure it out.
Traditional A/B testing focuses entirely on the first goal. By allocating equal traffic to all variants throughout a trial, it gathers clean, balanced data. However, as noted by Williams et al. and the authors of Comparison Lift, this incurs a massive opportunity cost: many participants are intentionally assigned to known sub-optimal groups just to prove statistical significance.
To fix this, we often turn to multi-armed bandits (MABs). Algorithms like Thompson Sampling dynamically route traffic away from losing arms and toward winning ones, minimizing cumulative regret (the welfare or revenue lost during the experiment).
But this creates a reverse problem. If an adaptive algorithm quickly funnels everyone to the best-performing arm, the sample sizes for the other arms remain tiny and highly biased. Without sufficient data from the alternative arms, we cannot calculate precise treatment effects, construct valid confidence intervals, or satisfy regulatory bodies.
This is not a failure of algorithm design; it is a fundamental mathematical limit. Recent literature establishes that adaptive experiments are governed by an inescapable Pareto frontier between estimation precision and participant regret.
The Intuition: Why Precision Requires Imbalance
To understand why this trade-off is mathematically ironclad, consider how we calculate the variance of a treatment effect estimate. If we want to measure the difference between a treatment arm ($A$) and a control arm ($B$), the variance of our estimate depends inversely on the sample sizes allocated to both:
$$\text{Var}(\hat{\tau}) \propto \frac{\sigma_A^2}{n_A} + \frac{\sigma_B^2}{n_B}$$
To minimize this variance (and maximize statistical power) for a fixed total number of participants $N$, we need to balance the allocations based on their internal variances—a principle known as Neyman Allocation. If the variances are equal, we must split the traffic exactly 50/50.
Now consider the goal of minimizing regret. If early data suggests that Arm A is superior to Arm B, a welfare-maximizing algorithm must immediately skew allocations, sending as close to 100% of participants to Arm A as possible ($n_A \to N, n_B \to 0$).
As $n_B$ shrinks, the term $\frac{\sigma_B^2}{n_B}$ blows up. By protecting current participants from an inferior treatment, we starve the experiment of the control data needed to precisely quantify how much better Arm A actually is. As formalized by Dai et al., minimizing outcome regret for individuals is fundamentally incompatible with minimizing the variance of a treatment effect estimate.
Statistical Precision (Neyman Allocation)
^
| * (50/50 Balanced Split)
| \
| \ The Pareto Frontier
| \
| \
| * (Adaptive Bandit Allocation)
+---------------------------------------> Participant Welfare (Min Regret)
Quantifying the Frontier
This relationship can be precisely bounded. Wang et al. proved that across adaptive experimental designs, the product of the estimation error and the square root of cumulative regret is bounded below by a constant:
$$\text{Estimation Error} \times \sqrt{\text{Regret}} \ge C$$
You can choose where to sit on this curve, but you cannot push past it. If you want to cut your estimation error in half, you must accept a proportional surge in cumulative regret.
This boundary manifests across different domains:
- Dynamic Pricing: In revenue management, Simchi-Levi et al. show a constant bound limits the product of causal estimation error (learning price elasticity) and revenue regret. You cannot maximize revenue during a test while perfectly mapping the demand curve.
- Contextual Personalization: When estimating Conditional Average Treatment Effects (CATE) for personalized policies, Li et al. demonstrate that this trade-off is governed by a minimax lower bound that scales directly with the dimensionality of the patient feature space. More features mean more exploration is required, compounding the potential regret.
- Public Policy: Welle frames this as a concave trade-off between resource targeting (giving scarce public services to those currently in highest need) and statistical learning (randomizing allocation to prove the program works for future funding).
Real-World Complications: Trends and Networks
The frontier becomes even more restrictive when we move away from idealized, static environments.
Non-Stationary Trends
In the real world, treatment effects drift. Drug resistance develops in clinical trials, and user preferences shift on web platforms. Simchi-Levi et al. analyzed experimental design under linear trends, proving that tracking a moving target requires continuous, prolonged exposure to suboptimal treatments. To accurately model the trajectory of a changing effect, you must deliberately incur a higher welfare loss than you would in a static environment.
Network Interference
Standard experimental design assumes that treating Participant A has no effect on Participant B. In social networks or ride-sharing platforms, this assumption breaks down due to spillover effects. Zhang et al. and Wang et al. studied design-based bandits under network interference. They proved that network spillover tightens the mathematical constraints on simultaneous learning and optimization, forcing an even steeper trade-off between estimation precision and cumulative regret.
Ethical and Algorithmic Solutions
Because we cannot break the Pareto frontier, modern research focuses on building algorithms that allow practitioners to safely navigate it based on their specific ethical or operational constraints.
Shifting the Focus to Future Populations
While standard adaptive trials try to maximize the health outcomes of currently enrolled patients, Bayesian Uncertainty Directed (BUD) designs, introduced by Ventz et al., explicitly prioritize gathering information that reduces entropy for future patient populations. Similarly, Wu et al. developed Partial Likelihood Thompson Sampling (PLTS) for pandemic vaccine trials, creating a tunable framework where researchers can explicitly trade off the number of infections permitted during the trial against the speed of identifying a definitive winner.
Balancing Fairness and Subgroups
Pure optimization can lead to severe ethical imbalances. If an algorithm finds that a treatment works incredibly well for a majority subgroup, it may stop exploring alternatives for a minority subgroup. Chien et al. and Wei et al. introduce fairness considerations into adaptive experiments, ensuring that algorithms do not sacrifice the welfare or estimation precision of specific demographic groups for global optimization. To safely implement this, Li et al. propose dynamic subgroup identification strategies that use response-adaptive randomization to find the best treatments for specific profiles without losing statistical validity.
Decoupling Learning from Regulation
Regulators like the FDA often require strict, un-biased randomization to approve a treatment, which clashes with active learning. To bridge this gap, Klein et al. proposed a two-stage framework called Randomize First Augment Next (RFAN). By separating the strict randomization required for hypothesis testing from the active data collection needed to optimize personalized policies, they demonstrate how to satisfy regulatory constraints while minimizing long-term welfare loss.
Ultimately, understanding the regret-inference trade-off changes how we evaluate algorithms. Success is no longer about finding a magical design that eliminates regret while maximizing power; it is about choosing an algorithm—like EXP3E (Wang et al.) or CLIP-OGD (Dai et al.)—that allows us to consciously and optimally place our experiment at the exact point on the Pareto frontier that our ethical commitments or business constraints demand.
Key Foundational Work
Bayesian Uncertainty Directed Trial Designs (2018) — contributed 2 concept elements to this direction.
Combining Dynamic A/B Experimentation and Recommender Systems in MOOCs (2016) — contributed 1 concept elements to this direction.