Abstract
Adaptive treatment assignment algorithms, such as bandit algorithms, are increasingly used in digital health intervention clinical trials. Frequently, the data collected from these trials is used to conduct causal inference and related data analyses to decide how to refine the intervention, and whether to roll-out the intervention more broadly. This work studies inference for estimands that depend on the adaptive algorithm itself; a simple example is the mean reward under the adaptive algorithm. Specifically, we investigate the replicability of statistical analyses concerning such estimands when using data from trials deploying adaptive treatment assignment algorithms. We demonstrate that many standard statistical estimators can be inconsistent and fail to be replicable across repetitions of the clinical trial, even as the sample size grows large. We show that this non-replicability is intimately related to properties of the adaptive algorithm itself. We introduce a formal definition of a "replicable bandit algorithm" and prove that under such algorithms, a wide variety of common statistical estimators are guaranteed to be consistent and asymptotically normal. We present both theoretical results and simulation studies based on a mobile health oral health self-care intervention. Our findings underscore the importance of designing adaptive algorithms with replicability in mind, especially for settings like digital health, where deployment decisions rely heavily on replicated evidence. We conclude by discussing open questions on the connections between algorithm design, statistical inference, and experimental replicability.
Introduction
In digital health, researchers use "bandit" algorithms to decide which intervention (like a push notification) to send to a user in real-time. After the trial ends, they use the collected data to answer critical questions: What was the average benefit of the treatment? Should we roll this out to the whole population?
The challenge is that these algorithms are adaptive—they learn and change their behavior based on the data they see. This paper identifies a "replicability crisis" inherent in standard bandit designs. Because algorithms like $\epsilon$-greedy make "hard" decisions (choosing the absolute best-looking arm), small amounts of noise can lead two identical trials to learn wildly different policies. Consequently, the statistical estimates derived from these trials don't match up, even with massive sample sizes.
Methodology: Replicable Bandit Algorithms
To address this, we introduce a formal mathematical definition of a Replicable Bandit Algorithm. Essentially, an algorithm is replicable if the policies it learns across independent repetitions of a trial become increasingly similar as the sample size grows.
The primary cause of non-replicability is the use of non-smooth, "greedy" decision rules. When an algorithm has a "winner-take-all" threshold, a tiny change in data can flip the treatment policy for thousands of people. We advocate for algorithms that use "soft" allocation functions to "stiffen" the algorithm against noise:
- Boltzmann Sampling: A "softmax" version of $\epsilon$-greedy that avoids hard thresholds.
- Smoothed Thompson Sampling: A modified version of the popular Bayesian algorithm that replaces the "argmax" indicator with a smooth function.
Key Results and Simulations
The paper evaluates these ideas using a high-fidelity simulation based on real-world data from the Oralytics pilot study—a mobile health intervention for oral self-care. This environment models complex human behaviors like app engagement and delayed treatment effects.
Our findings reveal several critical insights:
- Failure of $\epsilon$-greedy: In simulations with 1,000,000 individuals, $\epsilon$-greedy estimators still failed to concentrate, producing a bimodal distribution of results in trial runs (as seen in the figure above).
- Coverage Success: Under replicable algorithms (Boltzmann and smoothed Thompson sampling), 95% confidence intervals actually covered the true value roughly 95% of the time. In contrast, the $\epsilon$-greedy intervals had empirical coverage as low as 23.1%.
- Sample Variance Deception: In non-replicable settings, the variance calculated from a single trial can severely underestimate the true variability seen across trial repetitions, leading to a false sense of certainty.
Conclusion
While "hard" greedy algorithms might maximize rewards slightly better in the short term, they produce "fragile" data. Replicable bandits ensure the trial produces scientific evidence that can be reliably used for medical and policy decisions. Future work will further explore the formal trade-offs between regret minimization and replicability.
Summary updated