Abstract
Preventive healthcare programs often struggle with patient disengagement over time. We present Whittle Index based Q-Learning (WIQL), a model-free reinforcement learning algorithm designed for Restless Multi-Armed Bandits (RMAB) with unknown transition dynamics. WIQL enables organizations to optimally allocate limited intervention resources by learning patient behavior patterns online, significantly outperforming traditional heuristics in both synthetic and real-world maternal healthcare scenarios.
Introduction
In public health, maintaining long-term patient engagement is critical for the success of interventions like tuberculosis medication adherence or prenatal care. Organizations often employ automated systems (e.g., voice calls) to reach millions, but these systems suffer from high attrition. While manual interventions like home visits can re-engage patients, they are resource-intensive.
This challenge is formally a Restless Multi-Armed Bandit (RMAB) problem, characterized by:
- Restless Dynamics: Patient states (e.g., engaged vs. disengaged) change over time regardless of intervention.
- Resource Constraints: Only a small fraction $M$ of the total population $N$ can receive an intervention at any time step.
- Unknown Model: Transition probabilities are typically unknown and must be learned through interaction.
The WIQL Framework
The authors propose Whittle Index based Q-Learning (WIQL) to bridge the gap between theoretical RMAB solutions and practical, model-free applications. The core innovation lies in combining the Whittle Index—a powerful heuristic for RMABs—with Q-learning to estimate the value of actions without a prior transition model.
As illustrated in the state transition diagrams above, patients are modeled across three primary states:
- Self-motivated (S): High engagement; interventions provide little marginal benefit.
- Persuadable (P): Moderate engagement; these patients are most responsive to interventions.
- Lost Cause (L): Very low engagement; interventions are likely ineffective.
WIQL computes the difference between the Q-values of "active" (intervene) and "passive" (no intervention) actions. The algorithm prioritizes the top $M$ individuals with the highest difference, which the authors prove is equivalent to maximizing the joint Q-value across the population.
Experimental Evaluation
The researchers evaluated WIQL against several baselines, including "Random," "Myopic" (short-term greedy), and existing learning algorithms like "AB" and "Fu."
Synthetic Benchmarks
In controlled environments with circulant dynamics and restart models, WIQL consistently converged to the optimal policy (OPT). Unlike competing learning methods that often plateaued at sub-optimal rewards, WIQL demonstrated robust convergence even as the problem scale increased.
Real-World Application: Maternal Healthcare
Using data from ARMMAN, a non-profit in India, the authors simulated a large-scale intervention program. WIQL showed a clear advantage in maintaining total engagement scores compared to other strategies.
A key finding was WIQL's adaptability. In scenarios where patient behavior shifted mid-program (e.g., a sudden drop in responsiveness after 28 weeks), WIQL successfully detected the change in rewards and adjusted its prioritization strategy to recover engagement levels, whereas static or myopic models failed to adapt.
Discussion and Future Work
WIQL represents a significant step toward practical, adaptive resource allocation in healthcare. By identifying "persuadable" individuals, it ensures that limited staff time is spent where it has the highest impact.
However, challenges remain. The algorithm's convergence time increases with the complexity of the state space. Furthermore, the use of $\epsilon$-greedy exploration raises ethical questions in healthcare settings—randomly withholding a potentially life-saving intervention for "learning" purposes may not always be acceptable. Future research should focus on incorporating risk-aware constraints and handling heterogeneous intervention costs.
Summary updated