TL;DR
This paper introduces ReBandit, a novel online Reinforcement Learning (RL) algorithm designed for personalized mobile health interventions to reduce cannabis use among emerging adults. ReBandit leverages random effects and Bayesian priors to efficiently learn from noisy data and adapt to individual user heterogeneity, outperforming existing methods in simulations.
What Problem Does This Paper Address?
Cannabis use and cannabis use disorder (CUD) are significant public health challenges, particularly among emerging adults (EAs) aged 18-25. Despite the growing prevalence and associated health risks, there's a substantial treatment gap. Mobile health (mHealth) interventions, specifically Just-In-Time Adaptive Interventions (JITAIs), hold promise for delivering personalized support. However, developing effective JITAIs for mHealth faces several challenges:
- Limited Data: Sequential decision-making problems in mHealth often suffer from scarce data, making it difficult for RL algorithms to learn effectively.
- After-study Analysis: Evaluating the performance of RL algorithms after a study is crucial but can be complex.
- Autonomy and Stability: RL algorithms used in clinical settings must operate autonomously and stably in real-time, without requiring manual intervention during a study.
- Explainability: The decision-making process of RL algorithms needs to be interpretable for behavioral scientists and medical professionals.
- Delayed Effects: Interventions in mHealth can have delayed impacts on user behavior, which traditional RL algorithms, focused on immediate rewards, may not adequately capture.
- Reproducibility: The algorithms must be reproducible to ensure scientific rigor and facilitate implementation.
This paper aims to address these challenges by developing an RL algorithm tailored for personalized mHealth interventions.
Methods
The core of this paper is the ReBandit algorithm, an online RL approach designed to tackle the aforementioned mHealth challenges. Key methodological components include:
- Random Effects: ReBandit incorporates random effects into its model. This allows the algorithm to learn from the data of other participants in the study (population-level learning) while simultaneously personalizing interventions for each individual user. This helps overcome the "limited data" challenge by pooling information effectively.
- Informative Bayesian Priors: The algorithm utilizes pre-existing data to establish informative Bayesian priors. These priors act as a "warm-start," enabling ReBandit to learn more quickly and efficiently, especially in the early stages of a study.
- Bayesian Mixed Linear Model: ReBandit models the reward using a Bayesian Mixed Linear Model. This model combines population-level parameters (fixed effects) with individual-level parameters (random effects), allowing for adaptive pooling of data and personalized decision-making.
- Empirical Bayes and Optimization: The algorithm employs Empirical Bayes and optimization techniques to autonomously update its hyper-parameters online. This addresses the need for algorithmic autonomy and stability.
- Smooth Posterior Sampling: For action selection, ReBandit uses a modified posterior sampling algorithm called "smooth posterior sampling." This approach, which uses a generalized logistic function, ensures that randomization probabilities are smooth and concentrated, aiding in after-study analysis and off-policy evaluation.
- Engineered Reward: To account for delayed effects (C5), ReBandit uses an engineered reward function. This reward penalizes the action of sending an intervention message (action=1) based on a cost function that incorporates the standard deviation of observed rewards and a non-negative hyper-parameter. This encourages engagement while acknowledging potential negative impacts of frequent interventions.
Datasets
The paper uses data from the SARA [Rabbi et al., 2018] study to construct a simulation testbed. The SARA study involved an mHealth app for sustaining engagement in substance use data collection from participants, sharing a similar demographic (emerging adults) with the target population of the MiWaves pilot study. The authors cleaned and imputed the SARA data, resulting in a base dataset of 42 users. This data was used to fit Multinomial Logistic Regression (MLR) models for each user, which then served as the basis for simulating various environment variants.
Key Insights and Ideas
- Bridging the Gap: ReBandit effectively bridges the gap between fully personalized (each user gets their own model) and fully pooled (one model for everyone) approaches in RL for mHealth. By using random effects, it can leverage population data while still adapting to individual differences, which is crucial in heterogeneous populations.
- The Power of Priors: The use of informative Bayesian priors is highlighted as a critical factor for rapid and efficient learning in noisy mHealth environments, where data is often scarce and inconsistent.
- Balancing Engagement and Intervention Burden: The engineered reward function is a clever way to address the delayed effects of interventions. By introducing a cost for sending messages, the algorithm is incentivized to optimize for user engagement without overwhelming users with too many messages, which could lead to habituation or disengagement.
- Simulations as a Crucial Tool: The paper emphasizes the importance of robust simulation testbeds for evaluating RL algorithms in mHealth. By creating diverse environment variants (varying treatment effects and habituation levels), they could rigorously assess ReBandit's performance under different conditions.
Key Results
The paper presents simulation results comparing ReBandit against two baseline algorithms: a fully pooled Bayesian Linear Regression (BLR) algorithm and a random algorithm (π=0.5).
- Superior Performance: In most simulated environments, ReBandit performed equally well or better than both BLR and the random algorithm.
- Adaptability to Heterogeneity: The performance gap between ReBandit and BLR widened as population heterogeneity increased in the simulation environment. This demonstrates ReBandit's adeptness at adapting to diverse participant populations.
- Robustness Across Conditions: ReBandit showed strong performance across various levels of treatment effect (minimal, low, high) and habituation (low, high, none), indicating its robustness. For instance, in Table 1 (minimal and low treatment effect), ReBandit consistently achieved higher average total rewards per user compared to BLR and random, especially in scenarios with higher habituation (P=100%). Table 2 (high treatment effect) also showed ReBandit outperforming the other algorithms in most habituation scenarios.
Limitations and Future Work
- Simulation vs. Real-world: The study relies on simulations built upon existing data. While this is a common and necessary approach, the authors acknowledge that the simulation environments, particularly in how they artificially introduce treatment effects and habituation, may not fully capture the complexity and nuances of real-world user behavior and heterogeneity.
- Future Work: The authors plan to analyze the effectiveness of interventions in the upcoming MiWaves pilot study. Additionally, they aim to investigate the specific contributions of individual user data versus population data in learning the random effects parameters. They also express interest in exploring the impact of more pronounced heterogeneity in future studies.
Summary updated