Multi-Agent Deep Reinforcement Learning for Multi-Echelon Inventory Management

Xiaotian Liu, Ming Hu, Chunyi Peng, Yaodong YangProduction 2024 · DOI

TL;DR

This paper introduces a novel multi-agent deep reinforcement learning (MADRL) approach, HAPPO, to tackle complex multi-echelon inventory management problems. HAPPO significantly reduces overall costs and mitigates the bullwhip effect, outperforming traditional methods and single-agent RL. Crucially, it shows that a balanced approach where agents consider both their own costs and the system's overall costs leads to the best performance.

What Problem Does This Paper Address?

Modern supply chains are intricate networks with multiple entities (echelons) making independent inventory decisions. This decentralization, coupled with factors like demand uncertainty, lead times, and information asymmetry, can lead to the "bullwhip effect"—where demand variability amplifies as it moves upstream. This effect causes inefficiencies like excessive inventory or stockouts, harming economic performance. While deep reinforcement learning (DRL) has shown promise in optimizing individual inventory decisions, applying it to multi-agent, decentralized supply chains is challenging due to the non-stationary environment created by other agents' learning. Existing multi-agent RL approaches often assume a centralized "control tower" with full information, which is unrealistic in practice. This paper addresses the need for a MADRL approach that can handle decentralized decision-making with limited information sharing, specifically focusing on reducing costs and the bullwhip effect.

Methods

The core of this work is the application of Heterogeneous-Agent Proximal Policy Optimization (HAPPO), a multi-agent deep reinforcement learning (MADRL) algorithm. HAPPO utilizes a Centralized Training, Decentralized Execution (CTDE) paradigm. This means that during training, agents have access to global information (including other agents' observations and actions) to learn coordinated policies. However, during execution (real-world deployment), each agent makes decisions based solely on its own local observations, making the system practical.

Key methodological aspects include:

  • HAPPO: An extension of PPO designed for heterogeneous agents, ensuring that agents with different roles or characteristics can be effectively trained.
  • CTDE: A crucial mechanism that allows for effective learning in multi-agent systems by leveraging global information during training while maintaining decentralized operation.
  • Recurrent Neural Networks (RNNs) with GRUs: Used for actor and critic networks to capture temporal dependencies and historical information, which is vital for complex inventory dynamics.
  • Novel Reward Function: Instead of agents solely minimizing their own costs or the system's total costs, HAPPO uses a reward function that is a weighted combination of an agent's individual costs and the system's overall costs. This "balanced self-interest" is a key innovation.
  • Partially Observable Markov Game (POMG) Formulation: The supply chain problem is modeled as a POMG, a standard framework for multi-agent decision-making under uncertainty.

Datasets

The paper uses simulated data generated from various demand models to train and evaluate their proposed HAPPO algorithm and compare it against baselines.

  • Poisson Demands: A standard stochastic demand model.
  • Merton Jump Diffusion (MJD) Model: Used to simulate non-stationary customer demand, capturing sudden changes and heavy tails, which are common in real-world scenarios (e.g., pandemic-induced demand spikes).
  • Real-Life Demands: Data from the M5 forecasting competition, representing actual retail goods demand from Walmart.

Key Insights and Ideas

  • The Power of "Balanced Self-Interest": The most surprising finding is that agents performing best when they consider a combination of their own costs and the system's overall costs, rather than being purely self-interested or purely system-focused. This suggests that a degree of enlightened self-interest is optimal for decentralized agents.
  • Information Sharing is Key, Even in Training: The CTDE approach, with its upfront information sharing during training, significantly improves performance and reduces the bullwhip effect compared to methods that don't share information even during the learning phase.
  • MADRL for Complex Supply Chains: HAPPO demonstrates strong performance not only in simple serial supply chains but also in more complex network structures, validating its scalability.
  • Bullwhip Effect Mitigation: HAPPO effectively reduces the bullwhip effect, a critical problem in supply chain management, by learning coordinated ordering policies.
  • Robustness to Uncertainty: The approach shows robustness to various factors like lead time variations, cost coefficient changes, price discounts, and random shipping losses.

Key Results

  • Cost Reduction: HAPPO consistently achieved the lowest overall system costs across various scenarios, outperforming single-agent DRL (PPO) and heuristic policies (non-stationary base stock and (s, S) policies). In some cases, HAPPO reduced costs by over 1500% from initial states.
  • Bullwhip Effect Alleviation: HAPPO significantly reduced demand fluctuation amplification compared to the PPO baseline, indicating effective mitigation of the bullwhip effect. For instance, in one scenario, amplification decreased from 114% to 50% with HAPPO.
  • Optimal 'a' Value: The optimal balance parameter 'a' (weighting individual vs. system costs) was found to be between 0 and 1, confirming that neither extreme is optimal. Specific optimal 'a' values were identified for different supply chain configurations.
  • Convergence: Centralized HAPPO converged faster than HAPPO with full observation during training, which in turn converged faster than HAPPO with limited observation, highlighting the benefit of more information during training.

Limitations and Future Work

  • Computational Cost: While effective, MADRL training can be computationally intensive, especially for complex supply chains.
  • Generalizability of 'a': The optimal value of 'a' needs to be determined for each specific supply chain configuration, which might require extensive experimentation.
  • Future Work: The authors suggest applying MADRL to other multi-echelon inventory management problems, exploring other effective MADRL algorithms (like those following CTDE), and investigating scenarios with more restricted information sharing during training. They also provide extensible code for HAPPO to facilitate future research.

Summary updated

Abstract

We apply heterogeneous-agent proximal policy optimization (HAPPO), a multi-agent deep reinforcement learning (MADRL) algorithm, to the decentralized multi-echelon inventory management problems in both a serial supply chain and a supply chain network. We also examine whether the upfront-only information-sharing mechanism used in MADRL helps alleviate the bullwhip effect. Our results show that policies constructed by HAPPO achieve lower overall costs than policies constructed by single-agent deep reinforcement learning and other heuristic policies. Also, the application of HAPPO results in a less significant bullwhip effect than policies constructed by single-agent deep reinforcement learning where information is not shared among actors. Somewhat surprisingly, compared to using the overall costs of the system as a minimization target for each actor, HAPPO achieves lower overall costs when the minimization target for each actor is a combination of its own costs and the overall costs of the system. Our results provide a new perspective on the benefit of information sharing inside the supply chain that helps alleviate the bullwhip effect and improve the overall performance of the system. Upfront information sharing and action coordination in model training among actors is essential, with the former even more essential, for improving a supply chain’s overall performance when applying MADRL. Neither actors being fully self-interested nor actors being fully system-focused leads to the best practical performance of policies learned and constructed by MADRL. Our results also verify MADRL’s potential in solving various multi-echelon inventory management problems with complex supply chain structures and in non-stationary market environments.
BibTeX
@inproceedings{liu2024multi,
  author = {Xiaotian Liu and Ming Hu and Chunyi Peng and Yaodong Yang},
  title = {{Multi-Agent Deep Reinforcement Learning for Multi-Echelon Inventory Management}},
  booktitle = {Production},
  year = {2024},
  doi = {10.1177/10591478241305863},
}
Figure