Randomization as the Price of Admission: Collaborative Learning Across Micro-Randomized Trials Without Knowing How Different the Sites Are

Mobile health studies increasingly learn while they run. In a micro-randomized trial (MRT), a study that flips a biased coin hundreds of times per participant to decide whether to send a prompt, an online learning algorithm can adjust those coin biases as it learns who responds and when. Each study is small, though. HeartSteps V1 had 37 participants and the Affective Science smoking study had 68, so the learned policies stay noisy. Several health systems running similar interventions could pool what they learn. In principle, theory on collaborative bandits says this gives each site a speedup of roughly a square root of the number of sites. In practice, two obstacles block it. Optimal collaborative algorithms need to know in advance how different the sites are. A site whose population differs from the others has no reason to join if pooling makes its own policy worse.

We propose that both obstacles have the same solution, and that the solution is something MRTs already require. MRTs never allow the probability of any action to reach zero: HeartSteps clips it to $[0.1, 0.8]$. Because of this, every site keeps producing unbiased estimates of its own treatment effects for the whole study, with variances it can compute. That stream of guaranteed unbiased estimates lets the sites measure online how far apart they are. Each site can then choose its own degree of pooling so that collaboration provably does not hurt it. The forced randomization that looks like a tax on regret-optimal learning is what makes adaptive, incentive-compatible collaboration possible.

Core Claim

Elimination-style collaborative algorithms such as Collaborative Personalized Phased Elimination (CP-PE) need the population heterogeneity $\sigma^2$ as an input. On the optimal regret of collaborative personalized linear bandits proves minimax regret across three regimes. Full pooling is optimal when data are scarce and independent learning when data are abundant, but the switch is timed by a known $\sigma^2$. Once such an algorithm stops sampling an action, it can no longer tell whether that action's effect differs across agents.

An MRT policy with a probability floor $p_{\min}$ samples every action at least $p_{\min}$ of the time. It therefore keeps estimating every action's effect at variance $O(1/(p_{\min} n))$ per site, where $n$ is the number of decision points. Action-centering, as used in Personalized HeartSteps and in the weighted and centered least squares (WCLS) estimator of Data integration methods for micro-randomized trials, keeps these estimates unbiased even when the model of baseline behavior is wrong. The claim is that these two properties together let a coordinator do three things during the study, all without payments: estimate $\sigma^2$, give each site a personal pooling weight that recovers the CP-PE regret regimes up to log factors, and guarantee each site's individual rationality. Individual rationality here means the site's own regret and effect-estimation error are no worse than if it had run alone.

The mechanism is concrete. Let $\hat\theta_i$ be site $i$'s action-centered estimate of its $d$-dimensional effect parameter, with design-based covariance $V_i$. Let $\bar\theta_{-i}$ be the precision-weighted mean of the other sites' estimates, with covariance $\bar V_{-i}$. Site $i$ acts on

$$\tilde\theta_i = w_i \hat\theta_i + (1-w_i)\bar\theta_{-i}, \qquad \hat w_i = \arg\min_{w\in[0,1]} \; w^2\,\mathrm{tr}V_i + (1-w)^2\big(\hat b_i^2 + \mathrm{tr}\bar V_{-i}\big),$$

where $\hat b_i^2 = \max(0, \|\hat\theta_i-\bar\theta_{-i}\|^2 - \mathrm{tr}V_i - \mathrm{tr}\bar V_{-i})$ is an unbiased-before-truncation estimate of the squared gap between site $i$ and everyone else. That gap is the bias site $i$ would absorb by pooling. Averaging $\hat b_i^2$ across sites gives a method-of-moments estimate of $\sigma^2$.

The estimated error of $\tilde\theta_i$ is never larger than that of $\hat\theta_i$, since $w=1$ is always feasible. Individual rationality therefore holds by construction up to the error in $\hat b_i^2$. The theory has two tasks. It must bound that error using the randomization floor, and it must add a margin that turns the in-expectation guarantee into a high-probability one.

Research Program

The first part is theory, with regret measured against the best clipped policy that respects the daily budget. Under this comparator, clipping is part of the problem rather than a source of linear regret. We will prove three things for a Thompson-sampling or optimistic policy acting on $\tilde\theta_i$:

  1. Per-site regret matches the three CP-PE regimes up to logarithmic factors without $\sigma^2$ as input. The data-scarce regime gives joint regret $\tilde O(d\sqrt{mn})$ over $m$ sites, which is the linear speedup.
  2. The adaptivity cost is controlled by $p_{\min}$, because the error in $\hat b_i^2$ scales with $1/(p_{\min} n)$.
  3. With a confidence margin on $\hat b_i^2$, every site's regret is at most its solo regret plus a lower-order term, with high probability.

The second result explains why elimination-based collaboration cannot achieve adaptivity this way. The third answers the free-riding and defection concerns raised by One for One, or All for All and How to Incentivize Data-Driven Collaboration Among Competing Parties. In our setting the outside option is "run your own MRT." A site-specific pooling weight is the non-monetary reward that keeps every site above that option.

The second part handles the realities of MRTs:

  • Treatment-burden variable. We carry the exponentially decaying treatment-burden ("dosage") variable from Personalized HeartSteps as a moderator inside $\theta_i$.
  • Daily message budget. We enforce the budget through a budget-and-uniformity layer like the online risk-time sampler evaluated in Evaluation of the HeartSteps Online Sampling Algorithm.
  • Feedback from responses. We study the feedback loop that paper documents, in which responding to a prompt changes a participant's future eligibility.

The shared quantities are estimating-function summaries, meaning per-site effect estimates and their covariances. Raw data are never exchanged.

The third part addresses honesty, which Collaborative Causal Inference with Fair Incentives flags as an open weakness of incentive schemes. The shared algorithm sets every site's randomization probabilities. A site's reported treatment indicators can therefore be tested against those known probabilities, and its reported covariance against its design-based value. We will measure how much a misreporting site can gain before these audits catch it. We do not claim full strategy-proofness.

The relationship to prior work is as follows:

  • Collaborative Causal Inference with Fair Incentives gives individual rationality through perturbed rewards and Shapley-style valuation. It does so for a single static average treatment effect, and its Shapley computation becomes intractable beyond a handful of parties. We replace valuation with per-site shrinkage, which is linear in the number of sites and works for time-varying, moderated excursion effects. An excursion effect is the effect of sending versus not sending a prompt at one decision point.
  • Data integration methods for micro-randomized trials shows that borrowing across MRTs pays. Its precision-weighted estimator (PET-WCLS) gains 36.3% to 53.9% efficiency at $n=400$ in simulation. It is offline, assumes data are handed over freely, and depends on equal effects conditional on shared moderators. We make the borrowing online and individually rational, and we measure inequality between sites instead of assuming it away.
  • Learning with Limited Shared Information in Multi-agent Multi-armed Bandit ( Learning with Limited Shared Information in Multi-agent Multi-armed Bandit) gets speedup with individual rationality through a monetary broker. It works on context-free, homogeneous bandits. Our setting has no payments and has heterogeneous contexts.
  • Collaborative Heterogeneous Causal Inference Beyond Meta-analysis ( Collaborative Heterogeneous Causal Inference Beyond Meta-analysis) constrains our design. It shows that combining finished site estimates fails when sites under-cover parts of the population. We therefore pool parameters of a shared moderated effect model, not site-level averages.

Baselines

Independent learning. Each site runs the Personalized HeartSteps algorithm on its own: action-centered Thompson sampling with the burden variable and probability clipping, initialized with priors from V1-style pilot data. This is the outside option that individual rationality is measured against, and it is the system actually deployed in HeartSteps V2. The only reported comparison on Lacuna is in simulation, where it beat standard Thompson sampling by an average reward improvement of 29.753 (Personalized HeartSteps). Our method beats it if mean per-site regret drops in the data-scarce regime with no site worse off beyond the stated margin.

Naive pooling. One shared model is trained on all sites' data, the online analogue of the WCLS-Pooled estimator. It is the strongest competitor when sites are truly similar. It is also known to bias effect estimates under heterogeneity (Data integration methods for micro-randomized trials) and to stall convergence in federated Q-learning (On the Convergence Rates of Federated Q-Learning across Heterogeneous Environments). Our method beats it if it matches pooling at $\sigma^2\approx 0$ and avoids the individual-rationality violations pooling incurs as $\sigma^2$ grows.

CP-PE given the true $\sigma^2$. This is the strongest existing speedup method. We run it with its elimination step replaced by clipping, so that it respects the MRT design. It receives the oracle $\sigma^2$ in one arm of the comparison and a $\sigma^2$ estimated from pilot data in another. No reported numerical result was found on Lacuna for CP-PE; its paper confirms the regret regimes without simulator constants. Matching oracle CP-PE within a constant factor while being given no $\sigma^2$ counts as success.

Offline PET-WCLS integration. Sites learn independently and pool their estimates only after the trial. Its reported benchmark is 36.3% to 53.9% efficiency gain at $n=400$. We compare our method's end-of-study excursion-effect efficiency against it on the same simulator.

Ablation on the probability floor. We vary $p_{\min}$ from 0.1 down toward 0.01. If the floor is the key ingredient, the error in the $\sigma^2$ estimate and the rate of individual-rationality violations should both grow as $p_{\min}$ falls, roughly as $1/p_{\min}$.

Fairness of comparison. All methods share the same simulator seeds, site partitions, daily budget of 1.5 messages, clipping range, and an equal hyperparameter-tuning budget on held-out seeds.

Why not just…?

Why not just pool everything into one model? When sites are similar it is close to optimal, and we expect it to win slightly at $\sigma^2=0$, since it pays no shrinkage-estimation cost. That cost is the price of adaptivity and we will report it. Under heterogeneity, pooling bias is documented for MRT effect estimation and for federated learning. It also gives the most atypical sites the strongest reason to leave. The individual-rationality-violation metric measures directly how often that happens.

Why not just meta-analyze independent trials afterwards? Post-hoc meta-analysis, or the stronger PET-WCLS, recovers estimation efficiency but not the regret lost during the trial. Participants are treated by noisy solo policies for the whole study, which is the independent-learning regime in the CP-PE lower bounds. Our regret curves against offline PET-WCLS isolate this in-trial loss.

Why not just use a hierarchical Bayesian Thompson sampler that learns the between-site variance? This is the real threat. It is essentially our method without the design-based guarantees, and it may perform about as well in well-specified simulations. Our argument is that its $\sigma^2$ estimate comes from the outcome model. A misspecified baseline in step counts, which is common, inflates or deflates that estimate. The action-centered estimate does not have this problem. Our approach also yields an explicit per-site individual-rationality certificate. We will run the hierarchical sampler, with and without action-centering, under both well-specified and misspecified baselines. If it matches us under misspecification, our contribution reduces to the regret and individual-rationality analysis of that method, and we will say so.

Proposed Experiments

Main testbed. We will build a generative simulator from HeartSteps V1 following Personalized HeartSteps, fitting per-participant effect and burden parameters. Synthetic sites with $m$ from 2 to 32 and controlled heterogeneity $\sigma^2$ are created by shifting effect parameters along the moderators that matter. One such moderator is the sedentary-versus-active contrast, which produced about 2 versus about 1.1 messages per day in HeartSteps V2V3. In one condition we deliberately misspecify the baseline outcome model.

Metrics. We will measure:

  • per-site regret against the best budgeted clipped policy;
  • the fraction of sites whose regret exceeds their solo regret, with confidence intervals;
  • error in the $\sigma^2$ estimate over time;
  • post-study WCLS excursion-effect rMSE and 95% coverage;
  • adherence to the budget, with SeqRTS's achieved 1.486 messages per day (95% CI 1.365 to 1.607) as the reference.

Further checks. The CP-PE synthetic linear-bandit testbed checks whether our regret tracks the three regimes as $n$ grows. If the data are accessible, MARS (97 participants) and Affective Science (68 participants) serve as a two-site replay for the estimation side. There, our end-of-study estimates should match or exceed PET-WCLS efficiency without assuming conditional equality. A misreporting-site experiment measures the gain available to a site that fabricates treatment indicators before the randomization audit flags it.

Decisive test. Under baseline misspecification at moderate $\sigma^2$, the proposal needs two outcomes together: regret close to oracle CP-PE, and individual-rationality violations near zero. Naive pooling and the hierarchical sampler without action-centering should fail at least one of the two.