Abstract
Multi-agent reinforcement learning (MARL) is often hindered by non-stationary environments and inefficient exploration. This paper proposes the Multi-Agent coordinative Exploration algorithm with Prioritized Guidance (MAEPG). By integrating a centralized "Commander" critic with a reward-predicting "PrecoderNet," MAEPG synchronizes agent exploration and densifies sparse reward signals. Experimental results on the Gym Pendulum-v0 benchmark demonstrate that MAEPG converges more than twice as fast as the state-of-the-art MADDPG algorithm, particularly as the number of agents increases.
Introduction
In MARL, agents must learn optimal policies while the environment changes due to the actions of others. This leads to two primary challenges: redundant exploration, where agents fail to coordinate their search of the state-action space, and sparse rewards, which provide insufficient feedback for learning. While centralized training with decentralized execution (CTDE) methods like MADDPG address some stability issues, they often suffer from slow convergence and sample inefficiency.
Methods
The MAEPG framework introduces a hierarchical coordination structure to overcome these bottlenecks.
The architecture consists of three core components:
- The Commander (Centralized Critic): A single Q-value network coordinates multiple "soldier" agents. While each agent has its own local ActorNet, they share the Commander to evaluate joint actions, ensuring strategic alignment and reducing non-stationarity.
- PrecoderNet (Reward Guidance): This network predicts potential rewards for state-action pairs. This forecast serves as guidance ($\sigma$), modifying the environment reward ($r_{env}$) to create a denser signal: $r_{total} = r_{env} + \eta\sigma$. This effectively "shapes" the reward to provide clearer direction during early exploration.
- Enhanced Prioritized Experience Replay (PER): MAEPG improves standard PER by adding a "visiting time" factor ($\rho$) to the priority calculation. This prevents agents from over-sampling already well-explored regions, focusing instead on novel or high-error transitions.
Experimental Evaluation
The algorithm was evaluated using the Gym Pendulum-v0 environment. To test scalability and coordination, the authors configured the environment for both two-agent and three-agent collaborative tasks. Performance was compared against standard Deep Q-Networks (DQN) and Multi-Agent Deep Deterministic Policy Gradient (MADDPG).
Key Results
The experimental data highlights significant improvements in learning efficiency and stability across different agent counts.
- Convergence Speed: In the two-agent task, MAEPG reached convergence in roughly 110 episodes, whereas MADDPG required approximately 250 episodes.
- Scalability: The performance advantage of MAEPG became more pronounced in the three-agent scenario. The coordination provided by the Commander and PrecoderNet allowed the agents to maintain a steady learning trajectory where baselines struggled to scale.
- Stability: Unlike DQN, which showed high variance and slow progress, MAEPG demonstrated consistent upward trends in accumulative rewards, suggesting the reward guidance successfully mitigated the sparse reward problem.
Conclusion
MAEPG addresses the fundamental challenges of MARL by introducing predictive reward shaping and hierarchical coordination. By treating agent priority as a contribution weight for the centralized critic, the framework ensures that exploration is both diverse and efficient. Future research will explore the application of MAEPG in higher-dimensional environments and competitive multi-agent scenarios.
Summary updated