Pareto Optimal Trade-offs in Adaptive Experimental Design

This direction is merged into another direction: estimation-regret trade-off. Its last version stays readable and citable here.

Version 1 of 119 papers32 conceptsfrozen 2026-10-04

In a traditional randomized controlled trial (RCT), we split participants into two groups: one receives a treatment, and the other a placebo. This is statistically "pure" because it maximizes our ability to measure the difference between the two. However, it is often ethically fraught. If a drug is clearly saving lives halfway through a trial, continuing to give half the participants a placebo feels like a moral failure.

Adaptive experiments, often framed as Multi-Armed Bandits (MAB), attempt to solve this by shifting more participants toward the better-performing treatment as data comes in. But this creates a new problem: the more we prioritize the well-being of current participants (minimizing "regret"), the less we learn about the treatment's true effect for future patients (maximizing "statistical power").

Recent research has formalized this tension not as a binary choice, but as a Pareto frontier—a mathematical boundary where you cannot improve scientific certainty without sacrificing participant welfare, and vice versa.

The Fundamental Conflict: Welfare vs. Inference

The core of this research direction is the "Estimation-Regret Trade-off." In Multi-armed Bandit Experimental Design: Online Decision-making and Adaptive Inference, Chonghuan Wang and David Simchi-Levi prove that there is an unavoidable lower bound on the product of estimation error and the square root of regret.

Intuition suggests why: to know exactly how much better Treatment A is than Treatment B, you need a large sample of people on Treatment B—even if Treatment B is clearly inferior. If you stop giving people Treatment B to save them from poor outcomes, your "control" data becomes thin, and your estimate of the treatment effect becomes "noisy." Jessica Dai, Paula Gradu, and Christopher Harshaw highlight this in CLIP-OGD: An Experimental Design for Adaptive Neyman Allocation in Sequential Experiments, noting that maximizing participant outcomes is mathematically incompatible with minimizing the variance of a treatment effect estimate.

Rather than picking a single "best" algorithm, researchers are developing frameworks that allow experimenters to choose their position on this trade-off curve using tunable hyperparameters.

The Statistical Cost of Being Kind

While adaptive designs are ethically appealing, they come with a "statistical tax." Emmy Liu and colleagues point out in Statistical Consequences of Dueling Bandits that popular algorithms like Double Thompson Sampling can inflate false-positive rates. Because the algorithm "picks winners" early, the resulting dataset is skewed, making standard statistical tests unreliable.

This has led to the development of more robust inference methods. For instance, Han Wu and Stefan Wager proposed Partial Likelihood Thompson Sampling (PLTS) in Partial likelihood Thompson sampling to handle the complexities of survival analysis in vaccine trials, ensuring that we can identify the best treatment quickly without sacrificing the rigor needed for regulatory approval.

Beyond Binary Outcomes: Fairness and Targeting

The trade-off isn't just between "science" and "safety." It also involves fairness and resource allocation.

In Fair Adaptive Experiments, Waverly Wei, Xinwei Ma, and Jingshen Wang argue that standard power-optimizing designs can be unfair to certain subgroups. They propose frameworks that balance efficiency, fairness, and welfare simultaneously.

Similarly, Pim Welle explores this in the context of public policy in Learning treatment effects while treating those in need. Welle shows that the relationship between "targeting" (giving resources to those who need them most) and "learning" (randomizing to see if the program works) is a concave trade-off. Interestingly, policymakers can often get 90% of the targeting benefit while still maintaining high-quality evidence, provided they are willing to accept a small, calculated sacrifice in immediate service delivery.

By framing experimental design as a multi-objective optimization problem on a Pareto frontier, this research area provides a rigorous language for experimenters to navigate the difficult ethical and logistical choices inherent in modern data science and medicine.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_pareto_optimal_trade_offs_in_adaptive_ex_97fec2d2,
  title = {Pareto Optimal Trade-offs in Adaptive Experimental Design},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pareto-optimal-trade-offs-in-adaptive-experimental-design/rdr_97fec2d2162c462b945726f9507fa584},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_pareto_optimal_trade_offs_in_adaptive_ex_97fec2d2_v1,
  title = {Pareto Optimal Trade-offs in Adaptive Experimental Design},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/pareto-optimal-trade-offs-in-adaptive-experimental-design/rdr_97fec2d2162c462b945726f9507fa584/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}