Abstract
Mobile health (mHealth) interventions utilize Just-in-Time Adaptive Interventions (JITAIs) to promote healthy behaviors. However, training reinforcement learning (RL) agents for these systems is challenging due to the interaction burden on users and the lack of counterfactual data in historical logs. This paper presents a framework for training RL agents using a data-driven human simulator that integrates large-scale historical fitness data with psychological theories of memory and motivation. By incorporating a hard constraint on the frequency of interactions, the agent learns to deliver personalized, context-aware notifications. Results demonstrate that agents trained in this environment outperform context-blind baselines and develop sophisticated strategies, such as respecting user refractory periods and identifying high-utility contextual windows.
Introduction
The effectiveness of mobile health applications often hinges on the timing of notifications. Poorly timed nudges can lead to user annoyance and app abandonment—a challenge known as the interaction burden. Furthermore, traditional data-driven approaches struggle with the "counterfactual gap," as historical data rarely captures how a user would have reacted to an intervention that was never delivered. To address these issues, we develop a "human simulator" that serves as a proxy for real-world testing, allowing RL agents to explore and learn optimal policies offline before deployment.
Methodology
Human Simulator: Dynamic Bayesian Network
The core of our approach is a stochastic human simulator modeled as a Dynamic Bayesian Network (DBN). Unlike purely data-driven models, this simulator incorporates psychological insights to better capture the causal drivers of human behavior:
- Memory Accessibility: Modeled as an exponential decay function where a notification spikes accessibility, which then fades over time.
- Urge to Act: Modeled as a linear recovery process; after an activity (e.g., a run), the "urge" is low and gradually builds back up.
- Contextual Utility: Uses empirical data to determine how factors like weather and time of day affect the utility of the activity.
Reinforcement Learning with Constraints
For the learning component, we adapted the REINFORCE algorithm. A critical innovation is the inclusion of a hard constraint on the interaction frequency (e.g., a maximum of 14 notifications per week). This ensures the agent learns to "save" its notifications for high-impact moments rather than being "greedy," which is essential for maintaining user engagement in real-world scenarios.
Results
Performance and Policy Evolution
The framework was validated using a dataset of ~406,000 runs from over 10,000 users, matched with hourly meteorological data. The RL agent (R-REINFORCE) achieved significantly higher rewards compared to context-blind baselines, such as fixed-time or random intervals.
As training progressed, the agent's policy evolved from a naive distribution to a highly strategic one. As shown in the policy heatmaps, the agent initially sends notifications somewhat indiscriminately. By episode 16,000, it has learned to target "high-utility" windows (indicated by dark red circles), such as Sunday mornings with favorable weather. Additionally, the agent learned an emergent "refractory period" strategy, realizing that sending a notification immediately after a user finished a run was ineffective.
Discussion and Conclusion
By bridging psychological theory and large-scale empirical data, our simulator provides a robust environment for pre-training mHealth agents. This approach mitigates the risks of real-world trial-and-error while producing policies that are context-aware and respectful of user burden. While the "sim-to-real" gap remains a challenge, this methodology offers a promising path toward more effective and less intrusive digital health interventions. Future work will explore more advanced off-policy algorithms and long-term clinical validation.
Summary updated