Structured Actor-Critic for Managing Public Health Points-of-Dispensing

Yijia Wang, Daniel JiangarXiv 2018 · DOI · arXiv

Abstract

Public health organizations face the problem of dispensing treatments (i.e., vaccines, antibiotics, and others) to groups of affected populations through "points-of-dispensing" (PODs) during emergency situations, typically in the presence of complexities like demand stochasticity, heterogenous utilities, and limited storage. We formulate a hierarchical Markov decision process (MDP) model with two levels of decisions: an upper-level inventory planner and a lower-level dynamic dispensing problem. We propose a Structured Actor-Critic (S-AC) algorithm that leverages structure in both the policy (basestock) and value space (concavity). We prove convergence and demonstrate improved empirical performance through numerical analysis and a realistic case study for dispensing naloxone amidst the opioid crisis.

Introduction

During public health emergencies, the efficient distribution of medical supplies—such as vaccines, antibiotics, or overdose-reversal drugs like naloxone—is a critical logistical challenge. Public health organizations often utilize "Points-of-Dispensing" (PODs) to reach affected populations. However, managing these sites is complicated by several factors: demand is often unpredictable, storage capacity is limited, and different segments of the population may have varying priority levels (heterogeneous utilities).

This paper addresses these complexities by framing the problem as a hierarchical decision-making process. The goal is to develop a robust strategy that balances immediate needs with the necessity of maintaining inventory for future, potentially higher-priority, demands.

A Hierarchical Decision Framework

The researchers model the distribution problem as a hierarchical Markov Decision Process (MDP). This structure reflects the real-world division of labor in public health logistics:

  1. Upper-Level (Inventory Planner): At discrete time intervals (e.g., daily or weekly), a central planner decides how much stock to order and how much to allocate to local coordinators. These are "inventory decisions" characterized by "replenish-up-to" and "dispense-down-to" thresholds.
  2. Lower-Level (Local Coordinator): Between these inventory updates, a local coordinator must make real-time "dispensing decisions" as a random sequence of PODs arrives. They must decide how much of the current stock to dispense to each group, considering the group's specific priority or utility.
Timeline of hierarchical inventory and dispensing decisions.

The core challenge lies in the stochastic trade-off: satisfying a lower-priority demand immediately versus saving limited inventory for a potentially higher-priority demand later in the period.

Structured Actor-Critic (S-AC)

To solve this hierarchical MDP, the authors introduce the Structured Actor-Critic (S-AC) algorithm. Standard reinforcement learning (RL) often treats problems as "black boxes," learning policies from scratch. In contrast, S-AC incorporates known mathematical properties of optimal inventory policies directly into the learning architecture.

  • The Actor (Policy Structure): Instead of learning an arbitrary action distribution, the Actor is constrained to learn basestock thresholds. It assumes the optimal policy follows a "replenish-up-to" or "dispense-down-to" logic, which significantly narrows the search space.
  • The Critic (Value Structure): The Critic enforces discrete concavity (specifically L-concavity). This reflects the principle of diminishing marginal returns—each additional unit of inventory is generally less valuable than the previous one. By projecting the value function onto a concave space after each update, the algorithm remains mathematically consistent and stable.
  • Aggregation for Scalability: To handle high-dimensional real-world data, the authors use $k$-means clustering to group similar "information states" (such as regional overdose trends). This aggregation makes the problem computationally tractable without losing essential context.

Experimental Results and Case Study

The S-AC algorithm was evaluated against synthetic benchmarks and a real-world case study.

Synthetic Benchmarks: In tests with varying inventory capacities and information states, S-AC consistently reached over 99% optimality within 500–1,000 iterations. It significantly outperformed standard Actor-Critic, Policy Gradient, and Q-Learning methods, which were either slower to converge or plateaued at lower performance levels.

Pennsylvania Naloxone Case Study: Using five years of Emergency Medical Services (EMS) incident data from Open Data Pennsylvania (2021), the researchers simulated naloxone distribution in the five counties with the highest overdose rates. The S-AC approach achieved the highest total utility (measured in lives potentially saved or overdoses reversed) while maintaining moderate costs. It outperformed common heuristics like "First-Come-First-Served" or "Even Distribution" by effectively learning to prioritize high-risk areas.

Conclusion

By embedding mathematical structure—specifically inventory basestock policies and concave value functions—into an actor-critic framework, the S-AC algorithm offers a powerful tool for public health logistics. This "structured" approach acts as a regularizer, preventing the agent from exploring impractical strategies and leading to faster, more stable convergence. Future work may involve integrating shape-constrained deep neural networks to further enhance scalability and incorporating more complex inventory lead times.

Summary updated

Abstract

Public health organizations face the problem of dispensing treatments (i.e., vaccines, antibiotics, and others) to groups of affected populations through "points-of-dispensing" (PODs) during emergency situations, typically in the presence of complexities like demand stochasticity, heterogenous utilities (e.g., for vaccine distribution, certain segments of the population may need to be prioritized), and limited storage. We formulate a hierarchical Markov decision process (MDP) model with two levels of decisions (and decision-makers): the upper-level decisions come from an inventory planner that "controls" a lower-level dynamic problem, which optimizes dispensing decisions that take into consideration the heterogeneous utility functions of the random set of PODs. We then derive structural properties of the MDP model and propose an approximate dynamic programming (ADP) algorithm that leverages structure in both the policy and the value space (state-dependent basestocks and concavity, respectively). The algorithm can be considered an actor-critic method; to our knowledge, this paper is the first to jointly exploit policy and value structure within an actor-critic framework. We prove that the policy and value function approximations each converge to their optimal counterparts with probability one and provide a comprehensive numerical analysis showing improved empirical convergence rates when compared to other ADP techniques. Finally, we show how an aggregation-based version of our algorithm can be applied in a realistic case study for the problem of dispensing naloxone (an overdose reversal drug) via first responders amidst the ongoing opioid crisis.
BibTeX
@misc{wang2018structured,
  author = {Yijia Wang and Daniel Jiang},
  title = {{Structured Actor-Critic for Managing Public Health Points-of-Dispensing}},
  year = {2018},
  eprint = {1806.02490},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.02490},
}
Figure