Abstract
Multi-agent systems have a wide range of applications in cooperative and competitive tasks. As the number of agents increases, nonstationarity gets more serious in multi-agent reinforcement learning (MARL), which brings great difficulties to the learning process. Besides, current mainstream algorithms configure each agent an independent network, so that the memory usage increases linearly with the number of agents which greatly slows down the interaction with the environment. Inspired by Generative Adversarial Networks (GAN), this paper proposes an iterative update method (IU) to stabilize the nonstationary environment. Further, we add first-person perspective and represent all agents by only one network which can change agents' policies from sequential compute to batch compute. Similar to continual lifelong learning, we realize the iterative update method in this unified representative network (IUUR). In this method, iterative update can greatly alleviate the nonstationarity of the environment, unified representation can speed up the interaction with environment and avoid the linear growth of memory usage. Besides, this method does not bother decentralized execution and distributed deployment. Experiments show that compared with MADDPG, our algorithm achieves state-of-the-art performance and saves wall-clock time by a large margin especially with more agents.
Introduction
Multi-agent reinforcement learning (MARL) faces two significant hurdles as systems scale: environmental nonstationarity and excessive computational overhead. Nonstationarity, often called the "moving target" problem, occurs because agents learn simultaneously; from any single agent's perspective, the environment is constantly shifting as others change their behaviors. This instability often prevents convergence.
Furthermore, traditional MARL algorithms like MADDPG typically assign a unique neural network to each agent. This linear scaling of memory and computation creates a hardware bottleneck when dealing with dozens of agents. This research introduces IUUR (Iterative Update and Unified Representation) to address these challenges by stabilizing the learning signal and optimizing resource usage.
Methods
The IUUR framework employs two primary strategies to enhance MARL efficiency and stability:
- Iterative Update (IU): Drawing inspiration from the training dynamics of Generative Adversarial Networks (GANs), IU alternates training between agents. At any given time, one agent is designated as the "current learning agent" while others are "waiting." By fixing the policies of the waiting agents, the environment becomes stationary for the active learner, effectively reducing a complex multi-agent problem into a stable single-agent task.
- Unified Representation (UR): Instead of maintaining $N$ separate networks, IUUR utilizes a single shared network for all agents. By adopting a "first-person perspective"—where each agent's unique local observations are fed into the shared network—the system can perform batch computation. This allows actions for all agents to be calculated in a single forward pass, drastically accelerating environment interaction.
To maintain the strategies of "waiting" agents while updating the shared network for the active learner, the authors implement a "value fixing" method based on the Bellman Equation. This ensures that updates intended for one agent do not catastrophically interfere with the established policies of others.
Experimental Setup
The framework was evaluated using the Multi-Agent Particle Environment (MPE) across cooperative and competitive scenarios:
- Cooperative (Spread): Agents must navigate to target landmarks without colliding. Tests included Spread_3 (3 agents) and the more complex Spread_10 (10 agents).
- Mixed (Predator-Prey): A competitive scenario where predators coordinate to capture faster prey.
Key Results and Discussion
The IUUR algorithm was benchmarked against MADDPG, demonstrating significant improvements in both performance and efficiency:
- Performance Stability: In high-density scenarios like
Spread_10, IUUR achieved convergence where MADDPG struggled. The iterative approach provided the necessary stability for agents to learn effective cooperative behaviors. - Computational Efficiency: The transition to batch processing via Unified Representation yielded massive speedups. In the
Predator_3-prey_1scenario, training time was reduced from 6.4 hours to just 1.7 hours. Interaction time with the environment saw reductions of up to 75%. - Scalability: Unlike traditional methods, IUUR's memory footprint does not grow linearly with the number of agents, making it far more suitable for large-scale multi-agent systems.
While IUUR offers substantial benefits, it introduces a sensitivity to the update frequency hyperparameter ($K$). If agents are switched too frequently or too slowly, learning efficiency can degrade. Future research will focus on more robust "value fixing" techniques to further decouple agent updates within the shared architecture.
Conclusion
IUUR represents a significant step forward in making MARL practical for large-scale applications. By combining the stability of iterative updates with the efficiency of unified representation, the framework achieves state-of-the-art results while drastically lowering the barrier for computational resources.
Summary updated