Abstract
The task of decision-making under uncertainty is daunting, especially for problems which have significant complexity. Healthcare policy makers across the globe are facing problems under challenging constraints, with limited tools to help them make data driven decisions. In this work we frame the process of finding an optimal malaria policy as a stochastic multi-armed bandit problem, and implement three agent based strategies to explore the policy space. We apply a Gaussian Process regression to the findings of each agent, both for comparison and to account for stochastic results from simulating the spread of malaria in a fixed population. The generated policy spaces are compared with published results to give a direct reference with human expert decisions for the same simulated population. Our novel approach provides a powerful resource for policy makers, and a platform which can be readily extended to capture future more nuanced policy spaces.
Introduction
Malaria remains a devastating endemic disease in sub-Saharan Africa. While progress has been made, funding for prevention and control is leveling off, forcing policy makers to make difficult decisions about how to allocate limited resources. The "policy space"—the combination of various interventions like insecticide-treated nets (ITNs), indoor residual spraying (IRS), and vaccinations—is massive and complex. Traditionally, experts rely on intuition and limited manual simulations to decide which mix of tools will save the most lives for the least money. This research addresses the need for a systematic, automated way to explore this space to find the most cost-effective "one-shot" policies for a five-year intervention window.
Methods and Experimental Setup
The researchers frame the policy search as a Stochastic Multi-Armed Bandit (MAB) problem. In this setup, an "agent" chooses an action (a specific policy), observes a noisy reward from a simulator, and uses that information to decide its next move.
Three distinct agent-based strategies were implemented and compared:
- GP-ULCB (Gaussian Process Upper/Lower Confidence Bound): Uses Gaussian Process regression to model the policy space, sampling both high-performing areas (exploitation) and highly uncertain areas (exploration).
- Genetic Algorithm (GA): A biologically inspired optimizer that "evolves" better policies over generations using selection, crossover, and mutation.
- Batch Policy Gradient: A reinforcement learning approach using a neural network to approximate the reward function, updated via gradient descent.
The study utilized the OpenMalaria Platform, a sophisticated stochastic transmission simulator parameterized with data from the Rachuonyo South district in Western Kenya. Simulations were run on a representative human population of 100,000 using a high-performance computing cluster to allow for parallelized batch exploration.
Key Insights and Results
The primary metric used was Cost per DALY Averted ($C_{DA}$)—the cost to save one "healthy" year of life. Top-performing policies achieved $C_{DA}$ values ranging from approximately $25.50 to $30.90 USD.
A striking insight from the AI agents' findings challenges the current "nets-first" dogma. While many experts argue for maximizing ITN (net) coverage before considering other tools, the AI suggests that after reaching a certain threshold of net deployment, it is more cost-effective to shift funds toward a small proportion of indoor spraying (IRS) rather than simply buying more nets.
Despite having very different underlying logic, all three AI agents converged on the same top three emergent policy strategies. This suggests the discovered "peaks" in the policy space are robust. Furthermore, the agents were highly efficient, mapping the policy surface and identifying optimal zones after only 512 total simulations—a tiny fraction of the total possible policy combinations.
Limitations and Future Work
While promising, these results are currently specific to one district in Western Kenya and might not generalize across all of sub-Saharan Africa without further simulation. The study focused on "one-shot" 5-year policies, leaving room for future exploration of dynamic policies that adapt seasonally. Future work could also expand the action space to include targeted interventions or new vaccines and integrate deep learning to process visual data from malaria mapping projects.
Summary updated