Introduction
Pattern formation is a foundational challenge in swarm robotics and multi-agent systems, requiring a group of independent agents to organize into specific geometric configurations. While traditionally addressed through centralized control or decentralized methods with global knowledge, these approaches often fail to scale or rely on unrealistic assumptions about communication and sensing. This work proposes a decentralized deep reinforcement learning (DRL) architecture that enables hundreds of agents to form complex 2D shapes using only local sensor data and no explicit communication.
Problem Formulation
The task is framed as a decentralized partially observable Markov decision process (dec-POMDP). Agents are restricted to local observations, perceiving only a small $(2k+1) \times (2k+1)$ grid around themselves to identify teammates and obstacles. The challenge lies in solving the assignment problem (determining goal locations) and the trajectory planning problem (avoiding collisions) simultaneously under strict local sensing constraints.
Proposed Architecture
To overcome the limitations of local sensing, the authors introduce a novel learning framework characterized by three key components:
- Centralized Replay Memory: Although agents act independently, their experiences are pooled into a shared repository. This allows for collective learning from the successes and failures of all agents in the swarm.
- Centralized Target Network via Weight Averaging: To stabilize training and encourage a shared strategy, the "target network" is updated by averaging the weights of all individual agents' networks. This mechanism acts as a proxy for implicit communication, forcing agents to converge toward a unified policy that accounts for group dynamics.
- Homogeneity: All agents are identical and anonymous, ensuring the learned policy is robust to agent permutations.
Experimental Setup and Results
The system was evaluated in a $150 \times 150$ grid environment. Agents were trained on "random patterns" of up to 500 landmarks to learn general movement logic rather than specific shapes. Testing involved 550 agents forming structured, unseen patterns like a large "X".
Key Findings
- Generalization: Agents trained on random distributions successfully organized into complex, structured shapes without additional training.
- Performance and Scalability: The proposed method outperformed the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, which struggled with high agent densities.
- Efficiency: As shown in Figure 2, the proposed method achieves high success rates ($s$) and rewards ($r$) over time, reaching an approximate global shape quickly before fine-tuning positions.
Conclusion
This research demonstrates that decentralized agents can achieve complex global coordination through local sensing and weight-averaged target networks. The system successfully coordinated 550 agents, a significantly higher density than many contemporary multi-agent RL papers. Future work will explore dynamic transitions between multiple patterns and optimizing the computational trade-offs associated with visual field depth.
Summary updated