Optimizing Adaptive Notifications in Mobile Health Interventions Systems: Reinforcement Learning from a Data-driven Behavioral Simulator

Shihan Wang, Chao Zhang, Ben J. A. Kröse, Herke van Hoof

Abstract

Mobile health (mHealth) interventions utilize Just-in-Time Adaptive Interventions (JITAIs) to promote healthy behaviors. However, training reinforcement learning (RL) agents for these systems is challenging due to the interaction burden on users and the lack of counterfactual data in historical logs. This paper presents a framework for training RL agents using a data-driven human simulator that integrates large-scale historical fitness data with psychological theories of memory and motivation. By incorporating a hard constraint on the frequency of interactions, the agent learns to deliver personalized, context-aware notifications. Results demonstrate that agents trained in this environment outperform context-blind baselines and develop sophisticated strategies, such as respecting user refractory periods and identifying high-utility contextual windows.

Introduction

The effectiveness of mobile health applications often hinges on the timing of notifications. Poorly timed nudges can lead to user annoyance and app abandonment—a challenge known as the interaction burden. Furthermore, traditional data-driven approaches struggle with the "counterfactual gap," as historical data rarely captures how a user would have reacted to an intervention that was never delivered. To address these issues, we develop a "human simulator" that serves as a proxy for real-world testing, allowing RL agents to explore and learn optimal policies offline before deployment.

Methodology

Human Simulator: Dynamic Bayesian Network

The core of our approach is a stochastic human simulator modeled as a Dynamic Bayesian Network (DBN). Unlike purely data-driven models, this simulator incorporates psychological insights to better capture the causal drivers of human behavior:

  • Memory Accessibility: Modeled as an exponential decay function where a notification spikes accessibility, which then fades over time.
  • Urge to Act: Modeled as a linear recovery process; after an activity (e.g., a run), the "urge" is low and gradually builds back up.
  • Contextual Utility: Uses empirical data to determine how factors like weather and time of day affect the utility of the activity.
Overview of the RL framework for mHealth interventions, featuring a data-driven user simulator and a constrained RL agent.

Reinforcement Learning with Constraints

For the learning component, we adapted the REINFORCE algorithm. A critical innovation is the inclusion of a hard constraint on the interaction frequency (e.g., a maximum of 14 notifications per week). This ensures the agent learns to "save" its notifications for high-impact moments rather than being "greedy," which is essential for maintaining user engagement in real-world scenarios.

Results

Performance and Policy Evolution

The framework was validated using a dataset of ~406,000 runs from over 10,000 users, matched with hourly meteorological data. The RL agent (R-REINFORCE) achieved significantly higher rewards compared to context-blind baselines, such as fixed-time or random intervals.

Evolution of the notification policy over training episodes, showing the agent learning to target high-utility time windows.

As training progressed, the agent's policy evolved from a naive distribution to a highly strategic one. As shown in the policy heatmaps, the agent initially sends notifications somewhat indiscriminately. By episode 16,000, it has learned to target "high-utility" windows (indicated by dark red circles), such as Sunday mornings with favorable weather. Additionally, the agent learned an emergent "refractory period" strategy, realizing that sending a notification immediately after a user finished a run was ineffective.

Discussion and Conclusion

By bridging psychological theory and large-scale empirical data, our simulator provides a robust environment for pre-training mHealth agents. This approach mitigates the risks of real-world trial-and-error while producing policies that are context-aware and respectful of user burden. While the "sim-to-real" gap remains a challenge, this methodology offers a promising path toward more effective and less intrusive digital health interventions. Future work will explore more advanced off-policy algorithms and long-term clinical validation.

Summary updated

Abstract

Mobile health (mHealth) intervention systems can employ adaptive strategies to interact with users. Instead of designing such complex strategies manually, reinforcement learning (RL) can be used to adaptively optimize intervention strategies concerning the user’s context. In this paper, we focus on the issue of overwhelming interactions when learning a good adaptive strategy for the user in RL-based mHealth intervention agents. We present a data-driven approach integrating psychological insights and knowledge of historical data. It allows RL agents to optimize the strategy of delivering context-aware notifications from empirical data when counterfactual information (user responses when receiving notifications) is missing. Our approach also considers a constraint on the frequency of notifications, which reduces the interaction burden for users. We evaluated our approach in several simulation scenarios using real large-scale running data. The results indicate that our RL agent can deliver notifications in a manner that realizes a higher behavioral impact than context-blind strategies.
BibTeX
@article{wang2021optimizing,
  author = {Shihan Wang and Chao Zhang and Ben J. A. Kröse and Herke van Hoof},
  title = {{Optimizing Adaptive Notifications in Mobile Health Interventions Systems: Reinforcement Learning from a Data-driven Behavioral Simulator}},
  journal = {J. Medical Syst},
  year = {2021},
  doi = {10.1007/s10916-021-01773-0},
}
Figure