Context in Public Health for Underserved Communities: A Bayesian Approach to Online Restless Bandits

Biyonka Liang, Lily Xu, Aparna Taneja, Milind Tambe, Lucas Janson

In the world of public health, resources are almost always the bottleneck. Imagine an NGO in India, like ARMMAN, trying to improve maternal health by sending automated voice calls to thousands of pregnant women. Some women stop listening (non-adherence). The NGO can send healthcare workers to make live "troubleshooting" calls, but they only have the budget to reach about 2% of their beneficiaries each week.

Who gets the call? This is a classic resource allocation problem, but with a twist: the "arms" (beneficiaries) are "restless"—their state (adherence) changes over time regardless of whether you intervene. A new paper, "Context in Public Health for Underserved Communities: A Bayesian Approach to Online Restless Bandits," introduces a powerful new framework to solve this.

TL;DR

The authors present BCoR (Bayesian Learning for Contextual RMABs), an online reinforcement learning algorithm that uses hierarchical Bayesian modeling to share information across different beneficiaries and time. In real-world-inspired simulations for maternal health, BCoR achieved a 61% increase in engagement over existing state-of-the-art methods.

What Problem Does This Paper Address?

The authors tackle the Restless Multi-Armed Bandit (RMAB) problem in "data-scarce" environments. In public health, we face three brutal constraints:

  1. Unknown Dynamics: We don't know how a specific person will respond to a call until we try it.
  2. Tiny Budgets: The budget $B$ is much smaller than the population $N$. Most people are never called, meaning we never observe their "active" transition dynamics.
  3. Short Horizons: A pregnancy lasts about 40 weeks. Algorithms don't have thousands of steps to "warm up"; they must be effective almost immediately.

Existing methods like UCWhittle learn each person's behavior independently. If you have 500 people and only 10 calls per week, it takes forever to learn anything about the population. Furthermore, these methods often ignore context (like a mother's education or income) and non-stationarity (the fact that adherence patterns change as a pregnancy progresses).

Methods

The core innovation is BCoR, which combines Hierarchical Bayesian Modeling with Thompson Sampling.

  • Information Sharing: Instead of treating every beneficiary as a blank slate, BCoR uses a hierarchical prior. It assumes that mothers with similar backgrounds (contextual covariates) will behave similarly.
  • Passive-to-Active Transfer: It novelly uses data from "passive" states (when no call is made) to help predict how a person might react to an "active" intervention (a call).
  • Spline Regression: To handle non-stationarity, the model incorporates spline basis functions, allowing it to flexibly model how adherence naturally fluctuates over the 40-week timeline.
  • Thompson Sampling + Whittle Index: At each step, the model draws a sample from the posterior distribution of transition probabilities and uses the Whittle Index—a standard tool for RMABs—to pick the $B$ beneficiaries with the highest expected future value.

Datasets

The authors evaluated BCoR using two primary approaches:

  1. Synthetic Simulations: A suite of environments designed to test the model under "well-specified" conditions (where the data matches the model's assumptions) and "misspecified" conditions (e.g., the data is actually stationary, but the model is looking for time-trends).
  2. ARMMAN Data-Driven Simulator: This is the centerpiece of the evaluation. It uses real, anonymized covariate data from 24,011 beneficiaries in India. The simulator maps these real-world features (age, income, education) to internal risk estimates to create a highly realistic testing ground for public health interventions.

Key Insights and Ideas

  • Sharing is Caring (for Data): The "aha moment" is that in low-resource settings, individual learning is a luxury you can't afford. By sharing parameters across the population via context, the model can make educated guesses about a new beneficiary's behavior on Day 1.
  • Theory vs. Practice: The authors point out that many algorithms with "asymptotic regret guarantees" (they work perfectly if time goes to infinity) fail miserably in the short term. As seen in the results, some theoretical favorites actually perform worse than random selection when the time horizon is short.
  • Robustness to Misspecification: BCoR isn't just a "fair-weather" algorithm. Even when the researchers intentionally broke the model's assumptions (e.g., providing uninformative covariates), BCoR's performance remained stable, eventually learning to ignore the bad data.

Key Results

The empirical results are striking. In the ARMMAN-based simulations, BCoR consistently outperformed all competitors, including standard Thompson Sampling (TS) and UCWhittle.

Figure 2. Performance of various methods on the ARMMAN data-driven example described in Section 5.3 with N = 500, T = 40, and varying budget B, where all B ≤5% of N to reflect the magnitude of real-world budget constraints. Lines represent the time-averaged reward of each method averaged over 100 random seeds with the Random baseline subtracted out. Note, the grey line is an oracle approach with access to the true transitions. Error bars depict ±2 SEs. UCWhittle performs worse than random across all settings, which can sometimes occurs when the budget is relatively small and the time horizon is short, though it recovers over a longer time horizon; see

As shown in Figure 2 (above), BCoR (blue line) tracks much closer to the "Oracle" (the theoretical ceiling) than other methods. In the most resource-constrained setting ($B=10$, or 2% of the population), BCoR delivered a 61% improvement in cumulative engagement compared to the next best learning method.

Simulation results across various settings. BCoR shows superior finite-sample performance, especially when informative context is present.

Figure 1 demonstrates that BCoR's strength lies in its ability to leverage structure. In settings with informative covariates (a) and (b), the gap between BCoR and other methods is massive.

Limitations and Future Work

  • Computational Complexity: BCoR uses Hamiltonian Monte Carlo (HMC) for posterior updates. While this is fine for weekly updates (taking ~10 minutes for $N=400$), it might struggle with real-time, high-frequency applications.
  • Linearity: The current model assumes a linear relationship between covariates and the latent transition parameters. While the results show this is robust, future work could explore non-linear Bayesian models (like Gaussian Processes).
  • Deployment: While the simulator is built on real data, the next step is actual field deployment to see how these gains translate into clinical health outcomes for mothers and children.

Summary updated

Abstract

Public health programs often provide interventions to encourage program adherence, and effectively allocating interventions is vital for producing the greatest overall health outcomes, especially in underserved communities where resources are limited. Such resource allocation problems are often modeled as restless multi-armed bandits (RMABs) with unknown underlying transition dynamics, hence requiring online reinforcement learning (RL). We present Bayesian Learning for Contextual RMABs (BCoR), an online RL approach for RMABs that novelly combines techniques in Bayesian modeling with Thompson sampling to flexibly model the complex RMAB settings present in public health program adherence problems, namely context and non-stationarity. BCoR's key strength is the ability to leverage shared information within and between arms to learn the unknown RMAB transition dynamics quickly in intervention-scarce settings with relatively short time horizons, which is common in public health applications. Empirically, BCoR achieves substantially higher finite-sample performance over a range of experimental settings, including a setting using real-world adherence data that was developed in collaboration with ARMMAN, an NGO in India which runs a large-scale maternal mHealth program, showcasing BCoR practical utility and potential for real-world deployment.
BibTeX
@inproceedings{liang2025context,
  author = {Biyonka Liang and Lily Xu and Aparna Taneja and Milind Tambe and Lucas Janson},
  title = {{Context in Public Health for Underserved Communities: A Bayesian Approach to Online Restless Bandits}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i27.35039},
}
Figure