Uncovering the Varied Impact of Behavioral Change Messages on Population Groups

Jiaai Xu, Rada Mihalcea, Elena Frank, Srijan Sen, Maggie MakarMLHC 2023 · OpenReview

Abstract

Individuals such as medical interns who work in high-stress environments often face mental health challenges including depression and anxiety. These challenges are exacerbated by the limited access to traditional mental health services due to demanding work schedules. In this context, mobile health interventions such as push notifications targeting behavioral modification to improve mental health outcomes could deliver much needed support. In this work, we study the effectiveness of these interventions on subgroups, by studying the conditional average causal effect of these interventions. We design a two step approach for estimating the conditional average causal effect of interventions and identifying specific subgroups of the population who respond positively or negatively to the interventions. The first step of our approach follows existing causal effect estimation approaches, while the second step involves a novel tree-based approach to identify subgroups who respond to the treatment. The novelty in the second step stems from a pruning approach that deploys hypothesis testing to identify subgroups experiencing a statistically significant positive or negative causal effect. Using a semi-simulated dataset, we show that our approach retrieves affected subpopulations with a higher precision than alternatives while maintaining the same recall and accuracy. Using a real dataset with randomized push interventions among the medical intern population at a large hospital, we show how our approach can be used to identify subgroups who might benefit the most from interventions.

Introduction

Medical interns operate in high-pressure environments where traditional mental health support is often inaccessible due to grueling schedules. Mobile health (mHealth) interventions, such as push notifications, offer a scalable alternative. However, these interventions are typically evaluated based on their Average Treatment Effect (ATE), which can obscure critical variations in how different individuals respond. A "one size fits all" approach is rarely effective in mental health; an intervention that helps one person might be neutral for another or even detrimental to a third.

This research addresses the challenge of subgroup discovery: identifying which specific individual characteristics—such as medical specialty, baseline mental health, or demographics—predict a meaningful response to an intervention.

Methodology and Data

The researchers developed a two-step framework to estimate the Conditional Average Treatment Effect (CATE). First, they used a "T-learner" approach, training separate machine learning models for treated and control groups to predict outcomes. Second, they employed a novel decision tree-based approach to partition the population into subgroups.

A key innovation is the "honest" pruning strategy. Unlike standard decision trees that prune based on error reduction, this method uses an independent data slice to perform hypothesis testing on every leaf. This ensures that the identified subgroups are statistically distinct from their neighbors and that their observed effects are significantly different from zero, reducing noise and redundancy.

The method was validated using the Intern Health Study (IHS), a real-world micro-randomized trial involving over 1,500 medical interns. Participants provided baseline data (e.g., PHQ-9 depression scores) and wore Fitbits to track steps and sleep, while also reporting daily mood.

Participant flow and randomization for the Intern Health Study.

Key Findings: Beyond the Average

The study revealed that the impact of behavioral messages is highly nuanced. The researchers categorized interventions into three types: Mood, Steps, and Sleep.

Examples of behavioral change messages used in the study.

The Risk of Negative Effects

One of the most critical findings was that mood-focused notifications could have a negative impact. Interns with high baseline depression scores (PHQ > 9.5) who were not in Anesthesiology reported a deterioration in mood when receiving these messages (CATE = -0.694). This suggests that for vulnerable individuals, reminders about their mood might inadvertently increase stress or rumination. Conversely, Anesthesiology interns showed a significant positive response (CATE = 0.985).

Demographic Drivers in Sleep Interventions

While mood responses were tied to specialty and baseline health, sleep interventions were strongly moderated by demographics. Specifically, African American men saw a substantial increase in sleep duration—averaging 28.17 additional minutes per night—compared to the control group.

Precision in Subgroup Identification

In simulations, the "honest" pruning method achieved a precision of 0.45 in identifying affected subpopulations, significantly outperforming the classic-pruned baseline (0.39). This demonstrates that the method is more effective at finding the right subgroups rather than just more of them.

Discussion and Future Directions

While this approach provides a powerful tool for personalizing mHealth interventions, it has limitations. The study currently treats interventions as static snapshots, whereas mental health is dynamic. Future work could incorporate temporal changes as interns progress through their training. Additionally, while the effect of the messages is causal due to randomization, the subgroups themselves are observational; further research is needed to understand the underlying mechanisms behind these varied responses. Expanding the model to handle high-dimensional interventions, such as choosing between different message styles, remains a promising frontier for personalized mental health support.

Summary updated

Abstract

Individuals such as medical interns who work in high-stress environments often face mental health challenges including depression and anxiety. These challenges are exacerbated by the limited access to traditional mental health services due to demanding work schedules. In this context, mobile health interventions such as push notifications targeting behavioral modification to improve mental health outcomes could deliver much needed support. In this work, we study the effectiveness of these interventions on subgroups, by studying the conditional average causal effect of these interventions. We design a two step approach for estimating the conditional average causal effect of interventions and identifying specific subgroups of the population who respond positively or negatively to the interventions. The first step of our approach follows existing causal effect estimation approaches, while the second step involves a novel tree-based approach to identify subgroups who respond to the treatment. The novelty in the second step stems from a pruning approach that deploys hypothesis testing to identify subgroups experiencing a statistically significant positive or negative causal effect. Using a semi-simulated dataset, we show that our approach retrieves affected subpopulations with a higher precision than alternatives while maintaining the same recall and accuracy. Using a real dataset with randomized push interventions among the medical intern population at a large hospital, we show how our approach can be used to identify subgroups who might benefit the most from interventions.
BibTeX
@inproceedings{xu2023uncovering,
  author = {Jiaai Xu and Rada Mihalcea and Elena Frank and Srijan Sen and Maggie Makar},
  title = {{Uncovering the Varied Impact of Behavioral Change Messages on Population Groups}},
  booktitle = {MLHC},
  year = {2023},
  url = {https://openreview.net/forum?id=V3KVSFmiQ2},
}
Figure