Planning Multiple Epidemic Interventions with Reinforcement Learning

Anh L. Mai, Nikunj Gupta, Azza Abouzied, Dennis E. ShashaIJCAI 2023 · DOI

TL;DR

This paper tackles the complex challenge of planning optimal epidemic intervention strategies by framing it as a Markov Decision Process (MDP). They demonstrate that reinforcement learning, specifically Proximal Policy Optimization (PPO), can effectively learn intervention plans that outperform traditional hand-crafted baselines, offering a computationally supported approach for public health policymakers.

What Problem Does This Paper Address?

Designing effective epidemic intervention plans is a formidable task. Diseases vary in their spread dynamics, and the effectiveness of interventions like vaccines or mask mandates can fluctuate. Furthermore, our understanding of diseases evolves, requiring adaptable strategies. Traditional approaches often rely on simulating a limited set of predefined intervention plans, which fails to explore the vast space of potential strategies. Policy-makers need tools that can efficiently search for optimal plans that minimize disease burden and economic costs, especially when dealing with multiple, continuous interventions and complex disease models. This paper addresses this gap by proposing a reinforcement learning framework capable of handling these complexities.

Methods

The core of this work is the formulation of the epidemic planning problem as a Markov Decision Process (MDP). The agent (the RL algorithm) interacts with an EpiPolicy simulator (the environment) to learn an optimal policy.

  • MDP Formulation: The state ($S$) represents the distribution of the population across different disease compartments (e.g., susceptible, infected, recovered) and demographic groups. The action ($A$) is a set of applied interventions, each with continuous control parameters (e.g., mask compliance degree, vaccination administration rate). The transition function ($p(s_{t+1}|s_t, a_t)$) is provided by a deterministic compartmental disease simulator, and the reward ($r(s_t, a_t, s_{t+1})$) is the negative sum of intervention costs and disease burden costs.
  • Reinforcement Learning Algorithms: The authors explore Actor-Critic methods, specifically Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC). These methods are chosen for their ability to handle continuous state and action spaces, which is crucial for modeling realistic interventions.
  • Simulator: The EpiPolicy simulator is a key component, capable of simulating compartmental disease models (like SIR, SIRV, and a 15-compartment model) and programmatically-defined interventions with modifiable parameters and costs.

Datasets

The paper introduces a new benchmark consisting of six different environments, created by combining three state spaces (SIR, SIRV, C15) with two action spaces (A and B). These environments are designed to represent varying levels of disease and intervention complexity. The benchmark is motivated by and developed in consultation with public health officials.

Key Insights and Ideas

  • RL for Complex Intervention Planning: The paper demonstrates that reinforcement learning can effectively navigate the high-dimensional and continuous state and action spaces inherent in realistic epidemic planning scenarios, a feat difficult for traditional methods.
  • PPO's Robustness: PPO emerges as a more robust algorithm than SAC for this problem, requiring less hyperparameter tuning and demonstrating better performance across a range of environments. This is crucial for practical applications where rapid deployment of policy recommendations is needed.
  • Continuous Interventions Matter: The ability to model interventions with continuous parameters (rather than just binary choices) is essential for capturing the nuanced control policy-makers have and for discovering more effective strategies.
  • Smooth and Interpretable Policies: The learned policies from PPO are observed to be smooth and exhibit no drastic fluctuations, making them easier for policy-makers to understand and implement.

Key Results

  • PPO Outperforms Baselines: In all six benchmark environments, PPO significantly outperformed both SAC and hand-crafted baseline policies (Aggressive, Lax, Random) in terms of cumulative reward (minimizing cost).
  • Quantitative Performance: Table 4 shows that PPO achieved the highest maximum and mean cumulative rewards across all environments, with negative costs in the millions of dollars. For instance, in the C15-B environment, PPO's maximum reward was -45.2 million dollars, significantly better than the next best baseline (Lax at -166.6 million dollars).
  • SAC Performance: SAC performed comparably to PPO in simpler environments (SIR-A, SIRV-A) but underperformed in more complex ones, highlighting its sensitivity to hyperparameter tuning.
  • Training Efficiency: PPO training was significantly faster than SAC, with convergence occurring around 20,000 timesteps.

Limitations and Future Work

  • Hyperparameter Sensitivity: While PPO is more robust, both RL algorithms are sensitive to hyperparameter settings. The authors tuned hyperparameters on a single configuration and suggest that robust tuning across various parameter settings (like different transition rates) is necessary for real-world deployment.
  • Fully Observable State Assumption: The current work assumes a fully observable state. Future work could extend this to partially observable MDPs, where only partial state information is available.
  • Parameter Inference: The paper assumes that disease model parameters (like fatality or recovery rates) are known. Inferring these parameters (model calibration) is an active area of research and could be integrated.
  • Intervention Interactions: While the framework allows for multiple interventions, future work could explore more complex interactions between interventions.
  • Integration with Policy-Making: The authors emphasize the need for RL to be integrated into policy-making workflows, suggesting that faster training and more interpretable policies are key to this integration.

Summary updated

BibTeX
@inproceedings{mai2023planning,
  author = {Anh L. Mai and Nikunj Gupta and Azza Abouzied and Dennis E. Shasha},
  title = {{Planning Multiple Epidemic Interventions with Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/682},
}
Figure