Introduction
In modern smart factories, the efficiency of material handling depends on the seamless coordination of multiple autonomous robots. A common scenario involves "loading robots" retrieving materials from warehouses and "distribution robots" stationed at various workshops. To optimize throughput, these agents must converge at a dynamically determined rendezvous point that minimizes total travel distance. This challenge is formally defined as the Dynamic Multiple Sources to Single Destination (DMS-SD) problem.
Traditional approaches often rely on computationally intensive brute-force enumeration, which is ill-suited for the limited processing power of industrial IoT devices like Raspberry Pis. This paper presents a lightweight Deep Reinforcement Learning (DRL) framework that enables real-time, near-optimal coordination for robot swarms.
Proposed Framework
The researchers developed a DRL-based approach utilizing Proximal Policy Optimization (PPO). The system architecture is designed for scalability and efficiency in industrial environments, utilizing a Centralized Training, Decentralized Execution (CTDE) paradigm.
Key Methodological Innovations
- Centralized Training, Decentralized Execution (CTDE): Agents are trained in a global simulation environment where they have access to full state information. Once deployed, each robot operates independently using a local copy of the trained model, making decisions based on its specific observations.
- Homogeneous Model Sharing: To accelerate convergence and ensure unified swarm logic, all robots share the same neural network weights. This "shared brain" approach allows the system to scale to different swarm sizes without retraining.
- Potential-Based Reward Function: The reward mechanism guides robots toward the geometric centroid of the swarm. By framing the coordination problem as a target-seeking task toward a moving center, the complexity of multi-agent interaction is significantly reduced.
- Action Masking: To streamline the learning process, the system incorporates a hard-coded masking mechanism. It identifies and blocks invalid moves—such as collisions with walls or other robots—allowing the RL agent to focus exclusively on high-level strategic coordination rather than basic obstacle avoidance.
Experimental Evaluation
The framework was evaluated using NVIDIA Isaac Sim in grid-based factory layouts ranging from 50x50 to 70x70 units. The maps included a 5% density of static obstacles and a 2% density of dynamic obstacles, such as moving humans. Tests were conducted with swarms ranging from 2 to 12 robots.
Key Results
- Computational Efficiency: The DRL model generates movement decisions in milliseconds (~30ms on a desktop, <100ms on a Raspberry Pi 4), representing a 100x speedup over traditional non-learning solvers.
- Path Optimality: The paths generated by the DRL agents were found to be nearly identical to the mathematical optimum, typically differing by fewer than five steps across an entire trajectory.
- Robustness and Flexibility: The system effectively handles complex scenarios, such as escaping local optima caused by U-shaped obstacles. Furthermore, a model trained on a 12-robot swarm can successfully control smaller groups (e.g., 4 robots) using zero-padding in the state space, demonstrating high transferability.
Conclusion
This research demonstrates that a lightweight DRL framework with action masking and shared weights can solve complex multi-agent coordination problems in real-time. While currently validated on grid-based maps, future work will focus on extending these capabilities to continuous space navigation and incorporating additional constraints such as varying robot speeds and battery management.
Summary updated