Novel Exploration Techniques (NETs) for Malaria Policy Interventions

Oliver Bent, Sekou L. Remy, Stephen J. Roberts, Aisha Walcott-Bryant

Abstract

Public health officials in malaria-endemic regions face the challenge of optimizing resource distribution across various interventions under tight budget constraints. This study introduces a framework that utilizes AI agents to navigate the complex, stochastic landscape of malaria transmission simulations. By framing policy selection as a multi-armed bandit problem, we evaluate three exploration strategies: GP-ULCB, Genetic Algorithms, and Batch Policy Gradient. Our findings demonstrate that these AI agents consistently identify cost-effective intervention strategies that outperform traditional expert-led recommendations, highlighting the potential for AI to serve as a critical decision-support tool in global health policy.

Introduction

In sub-Saharan Africa, the fight against malaria is complicated by plateauing external funding and a massive "policy space" of possible intervention combinations. Traditionally, policies involving insecticide-treated nets (ITNs) and indoor residual spraying (IRS) are designed by human experts using simulation tools like OpenMalaria. However, the stochastic nature of these simulations—where identical inputs can yield varying outputs—makes manual optimization difficult. Human experts can only explore a fraction of the possible scenarios, often falling back on heuristics that may not be optimal.

Methodology: AI as a Policy Explorer

The researchers framed the search for optimal malaria policies as a Stochastic Multi-Armed Bandit problem. In this paradigm, an autonomous agent selects a "policy arm" (a specific combination of ITN and IRS coverage) and receives a reward based on the simulated health impact versus cost.

The interaction loop between the AI agent model and the OpenMalaria simulation environment.

Three distinct exploration strategies were compared:

  1. GP-ULCB (Gaussian Process Upper/Lower Confidence Bound): This approach uses Gaussian Process Regression to model the reward surface, balancing exploitation of known high-reward areas with exploration of uncertain regions.
  2. Genetic Algorithm (GA): A population-based approach where successful policies are "bred" and mutated to evolve better strategies over generations.
  3. Batch Policy Gradient: A reinforcement learning method using neural networks to approximate the reward function and propose new policy directions.

The agents were tested on a model of the Rachuonyo South district in Western Kenya. The primary optimization metric was the Cost per DALY Averted (Disability Adjusted Life Years), prioritizing the most efficient use of financial resources.

Results and Key Insights

The most significant finding was the discovery of a "threshold" effect. While traditional wisdom often suggests maximizing ITN coverage before introducing other interventions, the AI agents identified a point where it becomes more cost-effective to shift resources toward IRS rather than continuing to increase ITN deployment.

Comparison of the reward surface showing (a) current and expert human recommendations versus (b) the top three policies identified by the AI agents.

As shown in the reward surface mapping, all three AI algorithms converged on a similar set of top-performing policies. Specifically, the agents identified a strategy involving 60% ITN and 4% IRS coverage as highly cost-effective ($28.9 USD per DALY averted). In contrast, human expert recommendations typically favored high-coverage scenarios (e.g., 80% ITN and 90% IRS) which, while impactful, were less efficient in terms of cost per year of healthy life saved.

Discussion and Future Work

This research demonstrates that AI can uncover counter-intuitive strategies that human experts might overlook due to cognitive biases or the sheer complexity of the simulation landscape. By acting as a decision-support tool, these agents can help policymakers maximize health outcomes within limited budgets.

However, the current model is limited to two interventions and assumes static timing and homogeneous populations. Future work aims to expand the action space to include vaccines and drug treatments, and to utilize deep learning to incorporate geographic and demographic variations across sub-Saharan Africa.

Summary updated

Abstract

The task of decision-making under uncertainty is daunting, especially for problems which have significant complexity. Healthcare policy makers across the globe are facing problems under challenging constraints, with limited tools to help them make data driven decisions. In this work we frame the process of finding an optimal malaria policy as a stochastic multi-armed bandit problem, and implement three agent based strategies to explore the policy space. We apply a Gaussian Process regression to the findings of each agent, both for comparison and to account for stochastic results from simulating the spread of malaria in a fixed population. The generated policy spaces are compared with published results to give a direct reference with human expert decisions for the same simulated population. Our novel approach provides a powerful resource for policy makers, and a platform which can be readily extended to capture future more nuanced policy spaces.
BibTeX
@inproceedings{bent2018novel,
  author = {Oliver Bent and Sekou L. Remy and Stephen J. Roberts and Aisha Walcott-Bryant},
  title = {{Novel Exploration Techniques (NETs) for Malaria Policy Interventions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11410},
}
Figure