Deep reinforcement learning for the real-time inventory rack storage assignment and replenishment problem

Sander Teck, Tu-San Pham, Louis-Martin Rousseau, Pieter VansteenwegenEur. J. Oper. Res 2025 · DOI

Abstract

Robotic Mobile Fulfillment Systems (RMFS) have revolutionized e-commerce warehousing by using autonomous mobile robots (AMRs) to transport inventory racks to human pickers. This study introduces a Deep Reinforcement Learning (DRL) approach, specifically utilizing Proximal Policy Optimization (PPO), to solve the joint problem of real-time inventory rack storage assignment and replenishment. By integrating these two critical decisions, the proposed model reduces average robot cycle times by up to 5.4% and total travel distances by 6.3% compared to industry-standard heuristics.

Introduction

In modern fulfillment centers, efficiency is driven by the coordination of autonomous mobile robots. In an RMFS, robots carry entire racks to workstations for picking. Once a task is complete, the system must decide where to return the rack and whether it requires replenishment. Traditional methods rely on static heuristics, such as velocity-based storage or fixed replenishment thresholds, which often fail to account for dynamic factors like traffic congestion and future demand.

Layout of a typical Robotic Mobile Fulfillment System (RMFS) showing workstations, storage areas, and replenishment stations.

Integrated Problem Formulation

This research treats storage assignment and replenishment as a unified optimization problem. The goal is to minimize the average cycle time across the robot fleet. Unlike previous literature that often isolates these tasks, this integrated approach allows the system to make proactive decisions. For instance, a rack might be sent for replenishment earlier than strictly necessary if it is currently near a replenishment station and the queue is short, thereby avoiding future bottlenecks.

Robot travel paths for picking and replenishment tasks, illustrating the decision-making flow within the warehouse.

Methodology

The authors developed a DRL framework using the PPO algorithm. The state space is engineered to include high-dimensional data such as rack turnover rates, local traffic density, current stock levels, and SKU similarity within storage zones. To manage the complexity of the warehouse environment, storage locations are clustered into zones. The agent selects a target zone, and a "shortest leg" heuristic determines the specific slot. This "zoning" strategy balances optimization potential with computational efficiency, with nine zones identified as optimal for a 600-location setup.

Results and Discussion

The DRL agent was evaluated against several benchmarks, including random, nearest-open-slot, and velocity-based policies across 15 generated problem instances mimicking real-world configurations. The results demonstrate that the DRL policy consistently outperforms the best-performing heuristics. Key findings include:

  • Efficiency Gains: A 5.4% improvement in cycle time and a 4.1% reduction in total makespan.
  • Operational Optimization: A 6.3% reduction in total distance traveled, achieved by accounting for queuing times and traffic density rather than just physical distance.
  • Robustness: The agent showed strong generalizability when tested on unseen validation instances with varying SKU distributions.

Conclusion

By leveraging DRL, this study provides a more holistic and adaptive solution for RMFS management. The integration of storage and replenishment decisions allows for proactive warehouse flow optimization. Future research will focus on increasing the granularity of storage assignments and exploring multi-agent DRL to better manage high-congestion, bidirectional robot traffic.

Summary updated

BibTeX
@article{teck2025deep,
  author = {Sander Teck and Tu-San Pham and Louis-Martin Rousseau and Pieter Vansteenwegen},
  title = {{Deep reinforcement learning for the real-time inventory rack storage assignment and replenishment problem}},
  journal = {Eur. J. Oper. Res.},
  year = {2025},
  doi = {10.1016/j.ejor.2025.05.008},
}
Figure