reBandit: Random Effects based Online RL algorithm for Reducing Cannabis Use

Susobhan Ghosh, Yongyi Guo, Pei-Yao Hung, Lara N. Coughlin, Erin Bonar, Inbal Nahum-Shani, Maureen A. Walton, Susan A. Murphy

Abstract

The escalating prevalence of cannabis use, and associated cannabis-use disorder (CUD), poses a significant public health challenge globally. With a notably wide treatment gap, especially among emerging adults (EAs; ages 18-25), addressing cannabis use and CUD remains a pivotal objective within the 2030 United Nations Agenda for Sustainable Development Goals (SDG). In this work, we develop an online reinforcement learning (RL) algorithm called reBandit which will be utilized in a mobile health study to deliver personalized mobile health interventions aimed at reducing cannabis use among EAs. reBandit utilizes random effects and informative Bayesian priors to learn quickly and efficiently in noisy mobile health environments. Moreover, reBandit employs Empirical Bayes and optimization techniques to autonomously update its hyper-parameters online. To evaluate the performance of our algorithm, we construct a simulation testbed using data from a prior study, and compare against commonly used algorithms in mobile health studies. We show that reBandit performs equally well or better than all the baseline algorithms, and the performance gap widens as population heterogeneity increases in the simulation environment, proving its adeptness to adapt to diverse population of study participants.

Introduction

Mobile health (mHealth) interventions, specifically Just-In-Time Adaptive Interventions (JITAIs), represent a transformative approach to managing substance use disorders. By delivering support precisely when needed, these systems can significantly lower barriers to treatment. However, implementing reinforcement learning (RL) in this domain is fraught with difficulties. Clinical trials are typically brief (e.g., 30 days), data is often noisy due to self-reporting, and there is a fundamental tension between learning from a population (pooling) and tailoring to an individual (personalization). Furthermore, frequent interventions can lead to "user burden" or habituation, where the effectiveness of messages diminishes over time.

The reBandit Framework

The proposed reBandit algorithm addresses these challenges through a Contextual Bandit framework enhanced by Bayesian Mixed Linear Models. Unlike standard linear models that treat all users as identical or entirely independent, reBandit incorporates "random effects." It models a population-level "fixed effect" while allowing for individual-level "random effect" deviations. This structure enables the algorithm to "borrow strength" from the collective data of all participants to accelerate learning for new users, while still adapting to specific individual responses. This linear modeling approach also ensures interpretability, allowing researchers to understand the specific factors driving intervention decisions.

The MiWaves study design featuring 120 participants over 30 days with twice-daily decision points for RL-based interventions.

Key technical features include:

  • Empirical Bayes Updates: The algorithm autonomously tunes its own hyper-parameters (such as noise variance) online, reducing the need for manual intervention.
  • Smooth Posterior Sampling: By using a continuous probability function for intervention decisions rather than a hard rule, the algorithm ensures the resulting data is robust for post-hoc statistical analysis.
  • Reward Engineering: To mitigate habituation, the reward function includes a penalty for sending messages, encouraging the agent to intervene only when the predicted benefit is high.

Experimental Evaluation

As the primary MiWaves study is forthcoming, reBandit was evaluated using a simulation testbed derived from the SARA (Substance Use Research Assistant) study. The researchers created "generative clones" of 42 real-world users and tested the algorithm across 15 environments with varying treatment effects and habituation (HB) rates. Performance was measured against a standard Bayesian Linear Regression (BLR) with full pooling and a random intervention baseline.

Results and Discussion

The results demonstrate that reBandit consistently matches or exceeds the performance of standard BLR and random intervention strategies. Notably, the performance advantage of reBandit becomes more pronounced as population heterogeneity increases. In scenarios with high habituation and diverse user responses, reBandit’s ability to model individual deviations allowed it to significantly outperform pooled models. The "action centering" technique further bolstered robustness against model misspecification.

Performance comparison of reBandit against BLR and random baselines across various treatment effect and habituation scenarios.

As seen in the results table, reBandit maintains superior or competitive rewards across different levels of habituation (HB) and intervention probabilities (P). The algorithm demonstrated stability in autonomously updating its hyper-parameters in an online environment, a critical requirement for real-world deployment.

Conclusion

reBandit provides a robust, interpretable, and scalable solution for personalizing mHealth interventions. By effectively balancing population-level learning with individual-level adaptation, it overcomes the data scarcity and noise inherent in short-term clinical trials. The algorithm is slated for deployment in the March 2024 MiWaves pilot study, marking a significant step toward data-driven, personalized support for emerging adults struggling with cannabis use.

Summary updated

Abstract

The escalating prevalence of cannabis use, and associated cannabis-use disorder (CUD), poses a significant public health challenge globally. With a notably wide treatment gap, especially among emerging adults (EAs; ages 18-25), addressing cannabis use and CUD remains a pivotal objective within the 2030 United Nations Agenda for Sustainable Development Goals (SDG). In this work, we develop an online reinforcement learning (RL) algorithm called reBandit which will be utilized in a mobile health study to deliver personalized mobile health interventions aimed at reducing cannabis use among EAs. reBandit utilizes random effects and informative Bayesian priors to learn quickly and efficiently in noisy mobile health environments. Moreover, reBandit employs Empirical Bayes and optimization techniques to autonomously update its hyper-parameters online. To evaluate the performance of our algorithm, we construct a simulation testbed using data from a prior study, and compare against commonly used algorithms in mobile health studies. We show that reBandit performs equally well or better than all the baseline algorithms, and the performance gap widens as population heterogeneity increases in the simulation environment, proving its adeptness to adapt to diverse population of study participants.
BibTeX
@inproceedings{ghosh2024rebandit,
  author = {Susobhan Ghosh and Yongyi Guo and Pei-Yao Hung and Lara N. Coughlin and Erin Bonar and Inbal Nahum-Shani and Maureen A. Walton and Susan A. Murphy},
  title = {{reBandit: Random Effects based Online RL algorithm for Reducing Cannabis Use}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure