Abstract
Multi-agent reinforcement learning (MARL) for mapless navigation often struggles with non-stationarity and the "lazy agent" problem. Traditional Centralized Training and Decentralized Execution (CTDE) methods like VDN and QMIX impose strict monotonicity constraints that limit agent expressiveness. This paper introduces Multi-agent Global Dueling Q-learning (GDQ), a novel architecture that centralizes the state-value ($V$) rather than the action-value ($Q$). By decoupling local advantage from global state-value, GDQ enables effective coordination without restrictive mathematical constraints, leading to higher success rates and safer navigation in complex multi-robot environments.
Introduction
In multi-robot systems, navigating without a pre-existing map requires agents to learn behaviors that account for both static obstacles and the dynamic movements of teammates. Independent learning often fails because the environment becomes non-stationary as other robots update their policies. Conversely, fully centralized learning suffers from exponential complexity and the "lazy agent" phenomenon.
The current paradigm, Centralized Training and Decentralized Execution (CTDE), attempts to bridge this gap. However, state-of-the-art methods like VDN and QMIX rely on factorizing the joint team reward into individual rewards. To ensure consistency, these methods require that an individual's improvement always increases the team's total value (monotonicity). In dense navigation tasks, these constraints can prevent robots from learning the nuanced, cooperative maneuvers necessary for efficiency and safety.
Proposed Method: Multi-agent Global Dueling Q-learning (GDQ)
The authors propose GDQ, which leverages the Dueling Network architecture to bypass the factorization bottleneck. A standard Dueling Network estimates the State-Value ($V$)—the inherent value of being in a state—and the Advantage ($A$)—the relative benefit of specific actions.
GDQ introduces several key innovations:
- Centralized State-Value ($V_G$): Instead of factorizing the joint $Q$-value, GDQ uses a centralized network to estimate a global state-value ($V_G$) derived from individual robot state-values.
- Global Information Injection: During training, $V_G$ provides global context to the local update rule. Robots learn local actions ($A$) while evaluating their situation's quality based on the team's global state.
- Decoupled Execution: Because action selection depends solely on the local Advantage ($A$), robots do not need to share observations during real-time execution, satisfying the decentralized requirement.
By focusing on $V$ rather than $Q$, GDQ avoids the structural constraints of VDN and QMIX, allowing for more flexible and complex cooperative behaviors.
Experimental Setup
The GDQ framework was evaluated in two distinct environments:
- Cooperative Navigation: A benchmark task involving three agents and three landmarks in a 2D particle environment.
- Multi-Robot Mapless Navigation: A high-fidelity simulation in Unity/ROS using Turtlebot3 models. Robots were equipped with 2D Lidar and relative target tracking.
Experiments were conducted with 2, 4, and 8 robots to assess scalability and performance in increasingly crowded scenarios.
Results
GDQ consistently outperformed standard baselines, including IQL, VDN, and QMIX, across several metrics:
- Success Rate: In 4-robot tasks, GDQ reached a ~90% success rate significantly faster than QMIX (~80%).
- Scalability: With 8 robots, GDQ achieved a test reward of 24.3 ± 2.8, outperforming QMIX’s 18.2 ± 3.9.
- Safety: GDQ halved the collision rate compared to QMIX in the 8-robot scenario (17.3% vs. 34.2%).
- Efficiency: Agents trained with GDQ followed shorter, more direct paths to their targets.
The use of a centralized Double Deep Q-Network (DDQN) for the global state-value also helped mitigate overestimation bias, leading to more stable training curves.
Conclusion
GDQ represents a significant step forward in multi-robot mapless navigation by demonstrating that centralizing state-values is a viable and often superior alternative to action-value factorization. While the current work focuses on cooperative teams, future iterations could adapt the decoupled state-value stream for competitive environments. Additionally, bridging the "sim-to-real" gap through physical hardware testing remains an important next step for validating these findings in real-world robotics.
Summary updated