Abstract
Rising urban traffic demands smarter, cooperative control. Multi-agent deep reinforcement learning (MARL) offers a distributed approach to signal optimization, yet systematic comparisons of Q-network variants—DQN, DDQN, and Dueling DQN—remain scarce. We investigate how their architectural choices and update mechanisms affect multi-agent coordination. Theoretically, we dissect value-estimation bias and target-network coupling, clarifying how these factors shape joint policies. Empirically, we model a dynamic regional network in SUMO and benchmark the algorithms on stability, queue length, lane occupancy, mean speed, and throughput. Results show that dual-network decoupling accelerates training and raises network capacity, while the Dueling DDQN structure, by disentangling state value from advantages, curbs policy oscillation in multi-agent settings. The findings advance collaborative urban traffic control and provide transferable insights for multi-device scheduling.
Introduction
Urban traffic congestion is a persistent global challenge that traditional fixed-time signal control systems are increasingly ill-equipped to handle. The distributed nature of traffic networks—where local decisions at one intersection propagate to adjacent nodes—makes traffic signal control (TSC) an ideal candidate for Multi-Agent Deep Reinforcement Learning (MARL).
While Deep Q-Network (DQN) variants like Double DQN (DDQN) and Dueling DQN have demonstrated success in single-agent environments, their relative performance in complex, multi-agent traffic grids has not been rigorously compared. This study evaluates how different parameter update strategies and network architectures influence coordination and stability in a realistic regional traffic network.
Methodology
The research benchmarks four primary DRL architectures:
- DQN (Baseline): Uses a single network for action estimation, often leading to overestimation bias.
- Double DQN (DDQN): Addresses overestimation by decoupling action selection from evaluation using a target network.
- Dueling DQN: Modifies the internal architecture to separately estimate state value ($V(s)$) and action advantage ($A(s,a)$).
- D3QN (Dueling Double DQN): A hybrid architecture combining the benefits of both DDQN and Dueling DQN.
The experimental setup utilized the SUMO (Simulation of Urban MObility) package to model a three-junction arterial network. Traffic demand was varied to simulate peak and off-peak conditions, and performance was measured across metrics including queue length (JAMLEN), lane occupancy (OCC), mean speed (SPEED), and throughput (VOL).
Key Insights and Architectural Advantages
The Decoupling Advantage
The use of a target network in DDQN significantly stabilizes the training process. In MARL, the non-stationarity of the environment is exacerbated by agents constantly updating their policies. DDQN provides a more stable target for value estimation, preventing the "moving target" problem and allowing the multi-agent system to converge more reliably.
Disentangling Value from Advantage
The Dueling architecture is particularly suited for traffic control. In many traffic states, the overall congestion level (state value) is high regardless of the specific signal phase chosen. By separating the inherent value of a state from the specific advantage of an action, the network learns to identify critical decision points more effectively.
Results and Performance Comparison
The comparative analysis reveals distinct trade-offs between the architectures:
| Metric | D3QN (Hybrid) | DDQN | Dueling DQN |
|---|---|---|---|
| Queue Length | Best: $\le 30$m | Similar to D3QN | Peaks $\le 25$m |
| Mean Speed | ~9.5 m/s | ~10 m/s (high oscillation) | Best: 10 m/s (stable) |
| Convergence | Medium (~150 eps) | Slow | Fastest: (~100 eps) |
| Stability | Highest | Low (initial fluctuations) | Medium |
Dueling DQN acts as a "sprinter," achieving rapid convergence and high speeds, making it suitable for scenarios requiring fast adaptation. Conversely, D3QN serves as the "marathon runner," offering the highest long-term stability and congestion reduction in high-traffic scenarios, albeit at a higher computational cost.
Conclusion and Future Directions
The study demonstrates that architectural choices in DRL significantly impact the efficiency of urban traffic control. While D3QN provides the most robust performance for complex grids, its computational demands suggest a need for future research into model compression and edge deployment. Further work will also explore the resilience of these agents against unpredictable events like accidents or extreme weather conditions.
Summary updated