Robust Dynamic Material Handling via Adaptive Constrained Evolutionary Reinforcement Learning

Chengpeng Hu, Ziming Wang, Bo Yuan, Jialin Liu, Chengqi Zhang, Xin Yao

In the bustling environment of a modern "smart" factory, a fleet of Automated Guided Vehicles (AGVs) acts as the circulatory system, moving materials between workstations. But managing these fleets is a logistical nightmare: new tasks arrive unpredictably, vehicles break down, and strict deadlines (tardiness constraints) must be met.

A recent paper titled "Robust Dynamic Material Handling via Adaptive Constrained Evolutionary Reinforcement Learning" introduces ACERL, a new framework designed to make these systems both more efficient and more reliable.

TL;DR

ACERL is a novel reinforcement learning approach that optimizes AGV scheduling by combining gradient-free evolutionary strategies with an adaptive scenario sampler. It achieves near-perfect constraint satisfaction and superior efficiency (makespan) in dynamic environments, significantly outperforming existing state-of-the-art algorithms.

What Problem Does This Paper Address?

The authors tackle Dynamic Material Handling (DMH). Unlike static scheduling, DMH requires real-time decision-making in the face of "dynamic events" like machine failures or rush orders.

Current solutions face three major hurdles:

  1. Sparse Rewards: In a complex warehouse, you often don't know if a scheduling strategy was "good" until the very end of the shift when all tasks are finished. This makes traditional Reinforcement Learning (RL), which thrives on immediate feedback, very difficult to train.
  2. Strict Constraints: It’s not enough to be fast; you must avoid excessive task delays to keep the production line moving. Balancing speed (makespan) with these constraints usually requires tedious manual tuning of "penalty weights."
  3. Generalization: A policy trained on one warehouse layout or task distribution often fails when moved to a slightly different one or when noise is introduced.

Methods

ACERL (Adaptive Constrained Evolutionary Reinforcement Learning) introduces three key innovations to solve these issues:

  • Natural Evolution Strategies (NES): Instead of using standard backpropagation (which struggles with sparse rewards), ACERL maintains a population of "actors" (neural networks). It perturbs their parameters with Gaussian noise and updates the "parent" policy based on which perturbations performed best. This gradient-free approach is naturally more robust to the "all-or-nothing" feedback of DMH.
  • Intrinsic Stochastic Ranking (ISR): To handle constraints without manual weight tuning, the authors use a ranking system. Actors are grouped by the specific scenario they faced. Within those groups, they are ranked primarily by their ability to satisfy constraints and secondarily by their efficiency. This ensures the evolution "pressures" the policy to be feasible first and fast second.
  • Adaptive Instance Sampler (AIS): Training on every historical record equally is inefficient. ACERL uses an Upper Confidence Bound (UCB) approach—similar to how recommendation algorithms work—to dynamically select training scenarios. It focuses computational resources on instances where the current policy is struggling, ensuring a more robust final model.

Datasets

The researchers evaluated ACERL using the DMH-GYM simulator:

  • Training Set: 8 distinct problem instances (DMH-01 to DMH-08) drawn from different distributions.
  • Test Set: 8 unseen instances (DMH-09 to DMH-16) generated by adding noise to the training data.
  • Robustness Set: 40 additional unseen instances with varying levels of random perturbations to task arrival times.
  • Cross-Validation: A "leave-one-out" setup to test how well the model transfers to entirely different scenario distributions.

Key Insights and Ideas

  • Ranking Over Weighting: One of the biggest "aha" moments is that stochastic ranking (ISR) is far more effective than the standard "Lagrangian" methods used in constrained RL. By ranking individuals rather than adding a penalty term to a loss function, the algorithm avoids the numerical instability that often plagues constrained optimization.
  • The Power of Adaptive Training: The AIS component proves that what you train on matters as much as how you train. By focusing on "hard" scenarios, the model develops a more generalized understanding of the problem rather than just memorizing specific routes.
  • Evolutionary Advantage: In environments with high latency and sparse feedback (like a full warehouse shift), evolutionary strategies can actually converge faster and more reliably than popular policy gradient methods like PPO or MAPPO.

Key Results

ACERL was compared against five state-of-the-art RL/CRL algorithms (including MAPPO and RCPOM) and classic dispatching rules:

  • Constraint Satisfaction: ACERL achieved 100% constraint satisfaction on training instances and 97% on unseen test instances. For comparison, the previous best RL method (RCPOM) only managed around 60-63%.
  • Efficiency: It consistently achieved the best (lowest) makespan across nearly all test cases.
  • Speed: The trained policy can make a scheduling decision in just 2 milliseconds, easily meeting the requirements for real-time industrial deployment.
  • Robustness: Even when task arrival times were heavily "noised," ACERL maintained high performance, whereas traditional dispatching rules (like First-Come-First-Served) saw significant performance degradation.

Limitations and Future Work

While ACERL is highly effective, the authors noted some limitations:

  • Out-of-Distribution Gaps: In the leave-one-out cross-validation, performance dropped when the test instance was significantly different from the training cluster. This suggests that while the model is robust, it still benefits from seeing a diverse range of scenarios during training.
  • Complexity of State Space: The current model uses a specific encoding for AGV and task info. Scaling this to massive warehouses with hundreds of robots might require more hierarchical or multi-agent communication structures.

Future Directions: The authors suggest applying ACERL to other "real-world" constrained problems, such as vehicle routing and power grid optimization, where sparse rewards and strict safety constraints are equally prevalent.

Summary updated

Abstract

Dynamic material handling (DMH) involves the assignment of dynamically arriving material transporting tasks to suitable vehicles in real time for minimising makespan and tardiness. In real-world scenarios, historical task records are usually available, which enables the training of a decision policy on multiple instances consisting of historical records. Recently, reinforcement learning has been applied to solve DMH. Due to the occurrence of dynamic events such as new tasks, adaptability is highly required. Solving DMH is challenging since constraints including task delay should be satisfied. A feedback is received only when all tasks are served, which leads to sparse reward. Besides, making the best use of limited computational resources and historical records for training a robust policy is crucial. The time allocated to different problem instances would highly impact the learning process. To tackle those challenges, this paper proposes a novel adaptive constrained evolutionary reinforcement learning (ACERL) approach, which maintains a population of actors for diverse exploration. ACERL accesses each actor for tackling sparse rewards and constraint violation to restrict the behaviour of the policy. Moreover, ACERL adaptively selects the most beneficial training instances for improving the policy. Extensive experiments on eight training and eight unseen test instances demonstrate the outstanding performance of ACERL compared with several state-of-the-art algorithms. Policies trained by ACERL can schedule the vehicles while fully satisfying the constraints. Additional experiments on 40 unseen noised instances show the robust performance of ACERL. Cross-validation further presents the overall effectiveness of ACREL. Besides, a rigorous ablation study highlights the coordination and benefits of each ingredient of ACERL.
BibTeX
@article{hu2025robust,
  author = {Chengpeng Hu and Ziming Wang and Bo Yuan and Jialin Liu and Chengqi Zhang and Xin Yao},
  title = {{Robust Dynamic Material Handling via Adaptive Constrained Evolutionary Reinforcement Learning}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2025},
  doi = {10.1109/tnnls.2025.3582299},
}

Loading related content...

Figure