Estimating Time-Varying Direct and Indirect Causal Excursion Effects with Longitudinal Binary Outcomes

Jieru Shi, Zhenke Wu, Walter DempseyarXiv 2022 · DOI · arXiv

TL;DR

This paper introduces a robust statistical framework for analyzing Micro-Randomized Trials (MRTs) where participants are organized in clusters, such as medical interns in the same hospital. By accounting for "spillover" effects between peers and differences in how groups respond to interventions, the authors provide a more accurate way to measure how mobile health prompts influence binary behaviors like app engagement.

What Problem Does This Paper Address?

Mobile health (mHealth) apps often use "Just-In-Time Adaptive Interventions" (JITAIs)—think of a notification reminding you to log your food or check your mood. To design these effectively, researchers use Micro-Randomized Trials (MRTs), where individuals are randomized hundreds or thousands of times to receive (or not receive) an intervention.

However, existing statistical tools for MRTs with binary outcomes (like "did they complete the task: Yes/No?") make a risky assumption: that every participant is an island. In reality, people are often part of natural clusters (workplaces, schools, or medical specialties). This leads to two major challenges:

  1. Treatment Effect Heterogeneity: An intervention might be highly effective for surgeons but annoying for pediatricians.
  2. Interference: If one intern gets a notification to complete a survey, they might mention it to a colleague in the same residency, influencing that colleague's behavior even if the colleague wasn't "treated."

Ignoring these factors leads to "undercoverage"—statistical models that are overconfident and likely to produce false positives or inaccurate effect sizes.

Methods

The authors propose a new method called the Cluster-based Estimator of the Marginal Excursion Effect (C-EMEE). Key technical highlights include:

  • Novel Estimands: They define two types of effects:
  • Direct Effect: The impact of the intervention on the person who received it.
  • Pairwise Indirect Effect: The "spillover" impact on a peer within the same cluster.
  • Log Relative Risk Model: Instead of using odds ratios (which are common but difficult to average across groups), they use relative risk, which is "collapsible" and more scientifically intuitive for clinicians.
  • Robust Inference: They utilize a "sandwich" variance estimator with small-sample corrections (the Mancl and DeRouen method) to ensure the results remain valid even when the number of clusters is small.
  • Weighting: The method uses inverse probability weighting to handle "lagged" effects—measuring how a prompt today affects behavior several days from now.

Datasets

The primary real-world application is the Intern Health Study (IHS):

  • Scope: A 6-month MRT involving 1,565 medical interns.
  • Clustering: Interns were grouped into 321 residency institutions and 42 medical specialties.
  • Intervention: Weekly and daily randomization for mood, activity, or sleep notifications.
  • Outcome: Whether the intern completed a daily mood survey (binary).

The authors also used synthetic datasets in four simulation scenarios to stress-test the algorithm against varying cluster sizes and levels of interference.

Key Insights and Ideas

  • The "Aha" Moment on Variance: The authors show that when treatment effects vary by cluster, standard methods (like EMEE) drastically underestimate the standard error. In simulations, standard methods achieved only ~52% coverage for a 95% confidence interval, while C-EMEE maintained the full 95%.
  • The Negative Impact of Over-Prompting: In the Intern Health Study, the "direct effect" of a notification was actually negative. Sending a prompt made interns less likely to complete the survey the next day, particularly in the latter half of the six-month study. This suggests "notification fatigue" is a real hurdle for long-term JITAIs.
  • The Curse of the Horizon: Estimating "lagged" effects (the impact of a treatment on an outcome several time steps later) is significantly harder. The variance grows exponentially as the time window increases, a phenomenon known as the "curse of the horizon."

Key Results

  • Consistency and Normality: The authors mathematically proved that their estimator is consistent (converges to the truth) and asymptotically normal.
  • IHS Findings:
    • The marginal excursion effect was approximately -0.058 (p < 0.05), indicating a significant decrease in engagement following a prompt.
  • Engagement was most negatively affected for participants who already had a moderate completion rate (between 20% and 60%).
  • Indirect Effects: The study found limited evidence of spillover (indirect effects) between interns, suggesting that while they are in the same specialty, their survey-taking behavior remains relatively independent.

Limitations and Future Work

  • Binary Treatment Only: The current framework assumes a simple "Treatment vs. No Treatment" scenario. The authors suggest extending this to multi-level treatments (e.g., different types of notifications).
  • High-Dimensionality: As researchers add more "moderators" (factors like age, stress level, or location), the models become complex. Future work could involve regularization (like LASSO) to pick the most important factors.
  • Complex Networks: The paper assumes simple, non-overlapping clusters. Real-world social circles are often overlapping networks, which would require even more sophisticated interference modeling.

Summary updated

Abstract

Construction of just-in-time adaptive interventions, such as prompts delivered by mobile apps to promote and maintain behavioral change, requires knowledge about time-varying moderated effects to inform when and how we deliver intervention options. Micro-randomized trials (MRT) have emerged as a sequentially randomized design to gather requisite data for effect estimation. The existing literature (Qian et al., 2020; Boruvka et al., 2018; Dempsey et al., 2020) has defined a general class of causal estimands, referred to as "causal excursion effects", to assess the time-varying moderated effect. However, there is limited statistical literature on how to address potential between-cluster treatment effect heterogeneity and within-cluster interference in a sequential treatment setting for longitudinal binary outcomes. In this paper, based on a cluster conceptualization of the potential outcomes, we define a larger class of direct and indirect causal excursion effects for proximal and lagged binary outcomes, and propose a new inferential procedure that addresses effect heterogeneity and interference. We provide theoretical guarantees of consistency and asymptotic normality of the estimator. Extensive simulation studies confirm our theory empirically and show the proposed procedure provides consistent point estimator and interval estimates with valid coverage. Finally, we analyze a data set from a multi-institution MRT study to assess the time-varying moderated effects of mobile prompts upon binary study engagement outcomes.
BibTeX
@misc{shi2022estimating,
  author = {Jieru Shi and Zhenke Wu and Walter Dempsey},
  title = {{Estimating Time-Varying Direct and Indirect Causal Excursion Effects with Longitudinal Binary Outcomes}},
  year = {2022},
  eprint = {2212.01472},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.01472},
}
Figure