TL;DR
This paper introduces a robust statistical framework for analyzing Micro-Randomized Trials (MRTs) where participants are organized in clusters, such as medical interns in the same hospital. By accounting for "spillover" effects between peers and differences in how groups respond to interventions, the authors provide a more accurate way to measure how mobile health prompts influence binary behaviors like app engagement.
What Problem Does This Paper Address?
Mobile health (mHealth) apps often use "Just-In-Time Adaptive Interventions" (JITAIs)—think of a notification reminding you to log your food or check your mood. To design these effectively, researchers use Micro-Randomized Trials (MRTs), where individuals are randomized hundreds or thousands of times to receive (or not receive) an intervention.
However, existing statistical tools for MRTs with binary outcomes (like "did they complete the task: Yes/No?") make a risky assumption: that every participant is an island. In reality, people are often part of natural clusters (workplaces, schools, or medical specialties). This leads to two major challenges:
- Treatment Effect Heterogeneity: An intervention might be highly effective for surgeons but annoying for pediatricians.
- Interference: If one intern gets a notification to complete a survey, they might mention it to a colleague in the same residency, influencing that colleague's behavior even if the colleague wasn't "treated."
Ignoring these factors leads to "undercoverage"—statistical models that are overconfident and likely to produce false positives or inaccurate effect sizes.
Methods
The authors propose a new method called the Cluster-based Estimator of the Marginal Excursion Effect (C-EMEE). Key technical highlights include:
- Novel Estimands: They define two types of effects:
- Direct Effect: The impact of the intervention on the person who received it.
- Pairwise Indirect Effect: The "spillover" impact on a peer within the same cluster.
- Log Relative Risk Model: Instead of using odds ratios (which are common but difficult to average across groups), they use relative risk, which is "collapsible" and more scientifically intuitive for clinicians.
- Robust Inference: They utilize a "sandwich" variance estimator with small-sample corrections (the Mancl and DeRouen method) to ensure the results remain valid even when the number of clusters is small.
- Weighting: The method uses inverse probability weighting to handle "lagged" effects—measuring how a prompt today affects behavior several days from now.
Datasets
The primary real-world application is the Intern Health Study (IHS):
- Scope: A 6-month MRT involving 1,565 medical interns.
- Clustering: Interns were grouped into 321 residency institutions and 42 medical specialties.
- Intervention: Weekly and daily randomization for mood, activity, or sleep notifications.
- Outcome: Whether the intern completed a daily mood survey (binary).
The authors also used synthetic datasets in four simulation scenarios to stress-test the algorithm against varying cluster sizes and levels of interference.
Key Insights and Ideas
- The "Aha" Moment on Variance: The authors show that when treatment effects vary by cluster, standard methods (like EMEE) drastically underestimate the standard error. In simulations, standard methods achieved only ~52% coverage for a 95% confidence interval, while C-EMEE maintained the full 95%.
- The Negative Impact of Over-Prompting: In the Intern Health Study, the "direct effect" of a notification was actually negative. Sending a prompt made interns less likely to complete the survey the next day, particularly in the latter half of the six-month study. This suggests "notification fatigue" is a real hurdle for long-term JITAIs.
- The Curse of the Horizon: Estimating "lagged" effects (the impact of a treatment on an outcome several time steps later) is significantly harder. The variance grows exponentially as the time window increases, a phenomenon known as the "curse of the horizon."
Key Results
- Consistency and Normality: The authors mathematically proved that their estimator is consistent (converges to the truth) and asymptotically normal.
- IHS Findings:
- The marginal excursion effect was approximately -0.058 (p < 0.05), indicating a significant decrease in engagement following a prompt.
- Engagement was most negatively affected for participants who already had a moderate completion rate (between 20% and 60%).
- Indirect Effects: The study found limited evidence of spillover (indirect effects) between interns, suggesting that while they are in the same specialty, their survey-taking behavior remains relatively independent.
Limitations and Future Work
- Binary Treatment Only: The current framework assumes a simple "Treatment vs. No Treatment" scenario. The authors suggest extending this to multi-level treatments (e.g., different types of notifications).
- High-Dimensionality: As researchers add more "moderators" (factors like age, stress level, or location), the models become complex. Future work could involve regularization (like LASSO) to pick the most important factors.
- Complex Networks: The paper assumes simple, non-overlapping clusters. Real-world social circles are often overlapping networks, which would require even more sophisticated interference modeling.
Summary updated