TL;DR
This paper tackles the complex problem of managing inventory across a multi-node supply chain (warehouses and stores) with numerous products and finite capacities. They propose a novel multi-agent reinforcement learning (MARL) framework that enables parallel decision-making and effective coordination between different parts of the supply chain, outperforming traditional heuristic methods.
What Problem Does This Paper Address?
Modern retail supply chains are incredibly complex, involving thousands of products, numerous stores, and limited resources like warehouse space and truck capacity. The core challenge is to balance competing objectives: ensuring products are available to meet customer demand (avoiding stockouts) while minimizing waste from perishable goods and keeping operational costs low. Traditional methods often struggle with the sheer scale and combinatorial complexity of these problems, leading to suboptimal decisions. Specifically, managing multiple products simultaneously across multiple nodes with shared, finite capacities is a significant hurdle.
Methods
The authors propose a multi-agent reinforcement learning (MARL) approach. Each store and the warehouse are treated as independent agents. They utilize the Advantage Actor-Critic (A2C) algorithm, a popular deep reinforcement learning technique. A key innovation is the design of the state and action spaces, which are made independent of the number of products and stores, allowing for scalability. They also introduce a modified training procedure for the store agents that encourages smoother policies and faster convergence. For the warehouse agent, a binary action space is used, deciding whether to replenish a product or not. The training is conducted in a staged manner: first, store agents are trained assuming infinite warehouse capacity, and then the warehouse agent is trained in conjunction with the converged store policies.
Datasets
The experiments are based on a public dataset for brick-and-mortar stores (Kaggle). This dataset was adapted to include product dimensions and weights, and purchase data was transformed into forecasts and sales for 349 days. The authors created three experimental instances with 50, 220, and 1000 products, respectively, to evaluate their approach across different scales.
Key Insights and Ideas
- Scalability through MARL: Treating each store and the warehouse as independent agents within a MARL framework allows the system to scale to a large number of products and nodes without requiring a complete retraining of the entire system.
- Hierarchical but Decentralized: While the supply chain has a natural hierarchy (warehouse supplying stores), the RL approach is decentralized, with agents learning their own policies. This allows for flexibility in adding or removing nodes.
- Importance of Action Space Design: The design of the action space, particularly the quantization of replenishment quantities for stores and the binary decision for the warehouse, significantly impacts learning and performance.
- Transfer Learning Potential: The trained models demonstrate a remarkable ability to adapt to changes in the number of products and even the addition of new stores without requiring complete retraining, highlighting the practical applicability of the approach.
- Balancing Objectives: The RL agent learns to effectively balance the trade-offs between minimizing stockouts, reducing wastage, and managing replenishment costs, often outperforming traditional heuristics.
Key Results
- The proposed RL approach consistently outperforms traditional heuristic methods across all tested instances (50, 220, and 1000 products).
- In some cases, the RL policy even surpasses the "clairvoyant" heuristic (which uses perfect future sales data), suggesting that RL can find policies that adapt better to dynamic sales trends than fixed inventory level strategies.
- The system demonstrates good performance on test data, indicating its ability to generalize to unseen scenarios.
- Transfer learning experiments show that models trained on a smaller number of products can be effectively adapted to larger product sets, and the warehouse can handle new stores without full retraining.
Limitations and Future Work
- Forecast Accuracy: The performance of the RL agent is influenced by the accuracy of sales forecasts, which can vary significantly across different product sets.
- Warehouse Capacity Assumption: During the initial store training phase, the warehouse is assumed to have infinite capacity. This simplification is addressed in the subsequent joint training, but it's a limitation of the initial stage.
- Future Work: The authors suggest extending the problem to include different lead times for products from suppliers to the warehouse, further increasing the complexity and realism of the scenario. They also mention the potential for exploring more sophisticated action space designs and reward structures.
Summary updated