Introduction
Deep Reinforcement Learning (DRL) has emerged as a powerful tool for optimizing traffic signal control (TSC), offering the potential to significantly reduce urban congestion and commute times. However, transitioning these models from simulated environments to real-world city streets remains a significant challenge. Two primary obstacles hinder widespread adoption: robustness and generalizability.
In real-world deployments, sensors frequently fail due to hardware malfunctions or signal dropouts in "urban canyons," leading to incomplete data. Furthermore, training an RL agent for every specific city layout is computationally prohibitive. A truly effective TSC system must be resilient to data loss and capable of zero-shot transfer—the ability to perform effectively in a new, unseen environment without additional training. This paper introduces RGLight, a framework designed to bridge this gap by combining inductive graph representations with distributional reinforcement learning.
Methodology: The RGLight Framework
RGLight builds upon the foundations of Inductive Graph Reinforcement Learning (IGRL) but introduces a novel policy ensemble that integrates two distinct perspectives on reward estimation:
- Standard RL (The Mean): This component utilizes Graph Convolutional Networks (GCNs) to estimate the expected value of specific actions. By representing the traffic environment as a graph—where vehicles, lanes, and controllers are nodes—the model learns the underlying relationships between traffic entities rather than a fixed spatial map.
- Distributional RL (The Full Picture): Using Implicit Quantile Networks (IQN), RGLight models the entire distribution of potential returns. Instead of optimizing solely for the average outcome, the agent considers the full range of possibilities, including best- and worst-case scenarios. This distributional approach acts as a stabilizer, preventing the agent from overreacting to noisy data or outliers caused by sensor failures.
This dual-branch architecture allows RGLight to maintain fluid traffic flow. While standard RL models often suffer from high "switch rates" (changing lights too frequently and causing braking shockwaves), and pure distributional models may switch too slowly, RGLight finds a "Goldilocks" balance that optimizes throughput.
Experimental Setup and Datasets
The researchers evaluated RGLight using the SUMO (Simulation of Urban Mobility) micro-simulator across several scenarios:
- Training: The model was trained on diverse, randomly generated synthetic networks containing only 2 to 10 intersections.
- Synthetic Testing: Performance was tested on larger synthetic networks (up to 30 intersections) under "demand surges" where vehicle volume was doubled.
- Real-World Benchmarks: To test zero-shot transfer, the researchers deployed the model on two famous benchmarks: the rigid grid of Manhattan (75 lights) and the irregular, complex network of Luxembourg (22 lights).
Key Insights and Results
RGLight demonstrated significant improvements over traditional transportation heuristics (such as Greedy or Fixed-time) and state-of-the-art RL baselines.
Zero-Shot Generalizability
Despite being trained only on small, synthetic "toy" networks, RGLight successfully managed traffic in Manhattan and Luxembourg. It outperformed models that were specifically tuned for those environments, proving that its graph-based inductive bias captures universal "rules of the road" that transcend specific city geometries.
Resilience to Sensor Failure
The framework showed remarkable robustness. In the Manhattan simulation, even when 60% of sensor data was missing, RGLight maintained an average delay of 492 seconds. In comparison, the next best RL baseline saw delays spike to 653 seconds. By modeling the quantiles of traffic returns, the agent remains stable even when a sensor incorrectly reports zero speed due to a malfunction.
Handling Heavy Traffic
During simulated demand surges, RGLight reduced queue lengths by approximately 14% compared to previous state-of-the-art models like IGRL. This suggests that the distributional approach is particularly effective at managing the high-variance conditions typical of traffic congestion.
Limitations and Future Directions
While RGLight represents a major step forward, the authors note several areas for future research. The current model assumes lane-level knowledge of vehicles, which might not be available in total sensor blackouts. Additionally, as a "model-free" system, RGLight learns through trial and error; incorporating "model-based" RL could improve data efficiency. Finally, exploring the integration of Vehicle-to-Everything (V2X) communication could allow the system to leverage direct data from connected vehicles, further enhancing its predictive capabilities.
Conclusion
RGLight demonstrates that by combining graph-based representations with distributional reinforcement learning, it is possible to create traffic control systems that are both robust to real-world failures and portable across different urban environments. This framework paves the way for more reliable, scalable, and efficient smart city infrastructure.
Summary updated