Improving the Generalizability and Robustness of Large-Scale Traffic Signal Control

Tianyu Shi, François-Xavier Devailly, Denis Larocque, Laurent Charlin

Introduction

Deep Reinforcement Learning (DRL) has emerged as a powerful tool for optimizing traffic signal control (TSC), offering the potential to significantly reduce urban congestion and commute times. However, transitioning these models from simulated environments to real-world city streets remains a significant challenge. Two primary obstacles hinder widespread adoption: robustness and generalizability.

In real-world deployments, sensors frequently fail due to hardware malfunctions or signal dropouts in "urban canyons," leading to incomplete data. Furthermore, training an RL agent for every specific city layout is computationally prohibitive. A truly effective TSC system must be resilient to data loss and capable of zero-shot transfer—the ability to perform effectively in a new, unseen environment without additional training. This paper introduces RGLight, a framework designed to bridge this gap by combining inductive graph representations with distributional reinforcement learning.

Methodology: The RGLight Framework

RGLight builds upon the foundations of Inductive Graph Reinforcement Learning (IGRL) but introduces a novel policy ensemble that integrates two distinct perspectives on reward estimation:

  1. Standard RL (The Mean): This component utilizes Graph Convolutional Networks (GCNs) to estimate the expected value of specific actions. By representing the traffic environment as a graph—where vehicles, lanes, and controllers are nodes—the model learns the underlying relationships between traffic entities rather than a fixed spatial map.
  2. Distributional RL (The Full Picture): Using Implicit Quantile Networks (IQN), RGLight models the entire distribution of potential returns. Instead of optimizing solely for the average outcome, the agent considers the full range of possibilities, including best- and worst-case scenarios. This distributional approach acts as a stabilizer, preventing the agent from overreacting to noisy data or outliers caused by sensor failures.

This dual-branch architecture allows RGLight to maintain fluid traffic flow. While standard RL models often suffer from high "switch rates" (changing lights too frequently and causing braking shockwaves), and pure distributional models may switch too slowly, RGLight finds a "Goldilocks" balance that optimizes throughput.

Experimental Setup and Datasets

The researchers evaluated RGLight using the SUMO (Simulation of Urban Mobility) micro-simulator across several scenarios:

  • Training: The model was trained on diverse, randomly generated synthetic networks containing only 2 to 10 intersections.
  • Synthetic Testing: Performance was tested on larger synthetic networks (up to 30 intersections) under "demand surges" where vehicle volume was doubled.
  • Real-World Benchmarks: To test zero-shot transfer, the researchers deployed the model on two famous benchmarks: the rigid grid of Manhattan (75 lights) and the irregular, complex network of Luxembourg (22 lights).

Key Insights and Results

RGLight demonstrated significant improvements over traditional transportation heuristics (such as Greedy or Fixed-time) and state-of-the-art RL baselines.

Zero-Shot Generalizability

Despite being trained only on small, synthetic "toy" networks, RGLight successfully managed traffic in Manhattan and Luxembourg. It outperformed models that were specifically tuned for those environments, proving that its graph-based inductive bias captures universal "rules of the road" that transcend specific city geometries.

Resilience to Sensor Failure

The framework showed remarkable robustness. In the Manhattan simulation, even when 60% of sensor data was missing, RGLight maintained an average delay of 492 seconds. In comparison, the next best RL baseline saw delays spike to 653 seconds. By modeling the quantiles of traffic returns, the agent remains stable even when a sensor incorrectly reports zero speed due to a malfunction.

Handling Heavy Traffic

During simulated demand surges, RGLight reduced queue lengths by approximately 14% compared to previous state-of-the-art models like IGRL. This suggests that the distributional approach is particularly effective at managing the high-variance conditions typical of traffic congestion.

Limitations and Future Directions

While RGLight represents a major step forward, the authors note several areas for future research. The current model assumes lane-level knowledge of vehicles, which might not be available in total sensor blackouts. Additionally, as a "model-free" system, RGLight learns through trial and error; incorporating "model-based" RL could improve data efficiency. Finally, exploring the integration of Vehicle-to-Everything (V2X) communication could allow the system to leverage direct data from connected vehicles, further enhancing its predictive capabilities.

Conclusion

RGLight demonstrates that by combining graph-based representations with distributional reinforcement learning, it is possible to create traffic control systems that are both robust to real-world failures and portable across different urban environments. This framework paves the way for more reliable, scalable, and efficient smart city infrastructure.

Summary updated

Abstract

A number of deep reinforcement-learning (RL) approaches propose to control traffic signals. Compared to traditional approaches, RL approaches can learn from higher-dimensionality input road and vehicle sensors and better adapt to varying traffic conditions resulting in reduced travel times (in simulation). However, these RL methods require training from massive traffic sensor data. To offset this relative inefficiency, some recent RL methods have the ability to first learn from small-scale networks and then generalize to unseen city-scale networks without additional retraining (zero-shot transfer). In this work, we study the robustness of such methods along two axes. First, sensor failures and GPS occlusions create missing-data challenges and we show that recent methods remain brittle in the face of these missing data. Second, we provide a more systematic study of the generalization ability of RL methods to new networks with different traffic regimes. Again, we identify the limitations of recent approaches. We then propose using a combination of distributional and vanilla reinforcement learning through a policy ensemble. Building upon the state-of-the-art previous model which uses a decentralized approach for large-scale traffic signal control with graph convolutional networks (GCNs), we first learn models using a distributional reinforcement learning (DisRL) approach. In particular, we use implicit quantile networks (IQN) to model the state-action return distribution with quantile regression. For traffic signal control problems, an ensemble of standard RL and DisRL yields superior performance across different scenarios, including different levels of missing sensor data and traffic flow patterns. Furthermore, the learning scheme of the resulting model can improve zero-shot transferability to different road network structures, including both synthetic networks and real-world networks (e.g., Luxembourg, Manhattan). We conduct extensive experiments to compare our approach to multi-agent reinforcement learning and traditional transportation approaches. Results show that the proposed method improves robustness and generalizability in the face of missing data, varying road networks, and traffic flows.
BibTeX
@article{shi2023improving,
  author = {Tianyu Shi and François-Xavier Devailly and Denis Larocque and Laurent Charlin},
  title = {{Improving the Generalizability and Robustness of Large-Scale Traffic Signal Control}},
  journal = {IEEE Open Journal of Intelligent Transportation Systems},
  year = {2023},
  doi = {10.1109/ojits.2023.3331689},
}
Figure