Abstract
Public health organizations face the problem of dispensing treatments (i.e., vaccines, antibiotics, and others) to groups of affected populations through "points-of-dispensing" (PODs) during emergency situations, typically in the presence of complexities like demand stochasticity, heterogenous utilities, and limited storage. We formulate a hierarchical Markov decision process (MDP) model with two levels of decisions: an upper-level inventory planner and a lower-level dynamic dispensing problem. We propose a Structured Actor-Critic (S-AC) algorithm that leverages structure in both the policy (basestock) and value space (concavity). We prove convergence and demonstrate improved empirical performance through numerical analysis and a realistic case study for dispensing naloxone amidst the opioid crisis.
Introduction
During public health emergencies, the efficient distribution of medical supplies—such as vaccines, antibiotics, or overdose-reversal drugs like naloxone—is a critical logistical challenge. Public health organizations often utilize "Points-of-Dispensing" (PODs) to reach affected populations. However, managing these sites is complicated by several factors: demand is often unpredictable, storage capacity is limited, and different segments of the population may have varying priority levels (heterogeneous utilities).
This paper addresses these complexities by framing the problem as a hierarchical decision-making process. The goal is to develop a robust strategy that balances immediate needs with the necessity of maintaining inventory for future, potentially higher-priority, demands.
A Hierarchical Decision Framework
The researchers model the distribution problem as a hierarchical Markov Decision Process (MDP). This structure reflects the real-world division of labor in public health logistics:
- Upper-Level (Inventory Planner): At discrete time intervals (e.g., daily or weekly), a central planner decides how much stock to order and how much to allocate to local coordinators. These are "inventory decisions" characterized by "replenish-up-to" and "dispense-down-to" thresholds.
- Lower-Level (Local Coordinator): Between these inventory updates, a local coordinator must make real-time "dispensing decisions" as a random sequence of PODs arrives. They must decide how much of the current stock to dispense to each group, considering the group's specific priority or utility.
The core challenge lies in the stochastic trade-off: satisfying a lower-priority demand immediately versus saving limited inventory for a potentially higher-priority demand later in the period.
Structured Actor-Critic (S-AC)
To solve this hierarchical MDP, the authors introduce the Structured Actor-Critic (S-AC) algorithm. Standard reinforcement learning (RL) often treats problems as "black boxes," learning policies from scratch. In contrast, S-AC incorporates known mathematical properties of optimal inventory policies directly into the learning architecture.
- The Actor (Policy Structure): Instead of learning an arbitrary action distribution, the Actor is constrained to learn basestock thresholds. It assumes the optimal policy follows a "replenish-up-to" or "dispense-down-to" logic, which significantly narrows the search space.
- The Critic (Value Structure): The Critic enforces discrete concavity (specifically L-concavity). This reflects the principle of diminishing marginal returns—each additional unit of inventory is generally less valuable than the previous one. By projecting the value function onto a concave space after each update, the algorithm remains mathematically consistent and stable.
- Aggregation for Scalability: To handle high-dimensional real-world data, the authors use $k$-means clustering to group similar "information states" (such as regional overdose trends). This aggregation makes the problem computationally tractable without losing essential context.
Experimental Results and Case Study
The S-AC algorithm was evaluated against synthetic benchmarks and a real-world case study.
Synthetic Benchmarks: In tests with varying inventory capacities and information states, S-AC consistently reached over 99% optimality within 500–1,000 iterations. It significantly outperformed standard Actor-Critic, Policy Gradient, and Q-Learning methods, which were either slower to converge or plateaued at lower performance levels.
Pennsylvania Naloxone Case Study: Using five years of Emergency Medical Services (EMS) incident data from Open Data Pennsylvania (2021), the researchers simulated naloxone distribution in the five counties with the highest overdose rates. The S-AC approach achieved the highest total utility (measured in lives potentially saved or overdoses reversed) while maintaining moderate costs. It outperformed common heuristics like "First-Come-First-Served" or "Even Distribution" by effectively learning to prioritize high-risk areas.
Conclusion
By embedding mathematical structure—specifically inventory basestock policies and concave value functions—into an actor-critic framework, the S-AC algorithm offers a powerful tool for public health logistics. This "structured" approach acts as a regularizer, preventing the agent from exploring impractical strategies and leading to faster, more stable convergence. Future work may involve integrating shape-constrained deep neural networks to further enhance scalability and incorporating more complex inventory lead times.
Summary updated