DenseLight: Efficient Control for Large-scale Traffic Signals with Dense Feedback

Junfan Lin, Yuying Zhu, Lingbo Liu, Yang Liu, Guanbin Li, Liang Lin

Abstract

Traffic Signal Control (TSC) aims to reduce the average travel time of vehicles in a road network, which in turn enhances fuel utilization efficiency, air quality, and road safety, benefiting society as a whole. Due to the complexity of long-horizon control and coordination, most prior TSC methods leverage deep reinforcement learning (RL) to search for a control policy and have witnessed great success. However, TSC still faces two significant challenges. 1) The travel time of a vehicle is delayed feedback on the effectiveness of TSC policy at each traffic intersection since it is obtained after the vehicle has left the road network. Although several heuristic reward functions have been proposed as substitutes for travel time, they are usually biased and not leading the policy to improve in the correct direction. 2) The traffic condition of each intersection is influenced by the non-local intersections since vehicles traverse multiple intersections over time. Therefore, the TSC agent is required to leverage both the local observation and the non-local traffic conditions to predict the long-horizontal traffic conditions of each intersection comprehensively. To address these challenges, we propose DenseLight, a novel RL-based TSC method that employs an unbiased reward function to provide dense feedback on policy effectiveness and a non-local enhanced TSC agent to better predict future traffic conditions for more precise traffic control. Extensive experiments and ablation studies demonstrate that DenseLight can consistently outperform advanced baselines on various road networks with diverse traffic flows.

Introduction

Optimizing traffic signals is a fundamental challenge in urban planning. While deep reinforcement learning (RL) has emerged as a powerful tool for this task, it is hindered by two primary obstacles. First, the ultimate performance metric—total travel time—is a delayed signal, only available after a vehicle completes its journey. This forces researchers to use biased proxy rewards like queue lengths, which may not align with global travel time minimization. Second, traffic is a highly interconnected system where local conditions are heavily influenced by non-local dynamics, yet most existing agents focus primarily on immediate neighbors.

Methodology

DenseLight addresses these limitations through two core innovations: a mathematically unbiased dense reward and a communication-enhanced agent architecture.

Ideal-Factual Distance Gap (IFDG) Reward

Instead of relying on heuristic proxies, DenseLight introduces the Ideal-Factual Distance Gap (IFDG). This reward measures the difference between the distance a vehicle could have traveled at its maximum speed and the distance it actually traveled during a specific time step. The authors provide a mathematical proof that minimizing the cumulative IFDG is equivalent to minimizing total travel time. Unlike travel time, IFDG can be calculated every second for every vehicle, providing the dense feedback necessary for stable RL training.

Non-local Enhanced TSC (NL-TSC) Agent

To capture global traffic dependencies, the NL-TSC agent utilizes spatial-temporal augmentation and a novel communication layer. The agent concatenates current observations with previous states to capture trends and incorporates 2D position encodings to maintain spatial awareness.

Architecture of the NL-TSC agent, showing the information fusion process and the Dense Communication Layer.

As illustrated in the figure above, the architecture features Dense Communication Layers (DCL). While traditional multi-agent RL often uses Graph Neural Networks (GNNs) limited to local neighborhoods, DCL enables communication across the entire network. To ensure scalability for large cities, DenseLight employs a low-rank matrix decomposition ($W = W_a \times W_b$), keeping the parameter count manageable even as the number of intersections grows.

Experimental Setup

The framework was evaluated on four real-world road networks using the CityFlow simulator:

  • Jinan12 (4x3 grid)
  • Hangzhou16 (4x4 grid)
  • NewYork48 (16x3 grid)
  • NewYork196 (28x7 grid)

Additionally, a high-variance synthetic flow dataset ($D_{JN12(f)}$) was used to test the model's robustness against sudden traffic surges typical of rush-hour conditions.

Results and Discussion

DenseLight consistently outperformed state-of-the-art baselines, including FRAP, MPLight, and CoLight, across all tested scenarios.

  • Significant Efficiency Gains: In the large-scale NewYork196 network, DenseLight achieved an average travel time of 803 seconds, representing a 20% reduction compared to the best-performing baseline (Advanced-CoLight at 1004 seconds).
  • Robustness to Fluctuations: The model maintained its performance edge in high-variance traffic tests, suggesting it is better equipped to handle non-linear traffic spikes than heuristic-based models.
  • Ablation Insights: Studies confirmed that both the IFDG reward and the non-local communication branch are essential; removing either component led to a marked decrease in performance.

Conclusion

DenseLight provides a scalable and theoretically grounded approach to large-scale traffic signal control. By bridging the gap between local observations and global network states with dense, unbiased feedback, it offers a significant step toward more efficient urban mobility. Future work will explore zero-shot generalization to new city layouts and the integration of heterogeneous traffic participants like pedestrians and emergency vehicles.

Summary updated

Abstract

Traffic Signal Control (TSC) aims to reduce the average travel time of vehicles in a road network, which in turn enhances fuel utilization efficiency, air quality, and road safety, benefiting society as a whole. Due to the complexity of long-horizon control and coordination, most prior TSC methods leverage deep reinforcement learning (RL) to search for a control policy and have witnessed great success. However, TSC still faces two significant challenges. 1) The travel time of a vehicle is delayed feedback on the effectiveness of TSC policy at each traffic intersection since it is obtained after the vehicle has left the road network. Although several heuristic reward functions have been proposed as substitutes for travel time, they are usually biased and not leading the policy to improve in the correct direction. 2) The traffic condition of each intersection is influenced by the non-local intersections since vehicles traverse multiple intersections over time. Therefore, the TSC agent is required to leverage both the local observation and the non-local traffic conditions to predict the long-horizontal traffic conditions of each intersection comprehensively. To address these challenges, we propose DenseLight, a novel RL-based TSC method that employs an unbiased reward function to provide dense feedback on policy effectiveness and a non-local enhanced TSC agent to better predict future traffic conditions for more precise traffic control. Extensive experiments and ablation studies demonstrate that DenseLight can consistently outperform advanced baselines on various road networks with diverse traffic flows. The code is available at https://github.com/junfanlin/DenseLight.
BibTeX
@inproceedings{lin2023denselight,
  author = {Junfan Lin and Yuying Zhu and Lingbo Liu and Yang Liu and Guanbin Li and Liang Lin},
  title = {{DenseLight: Efficient Control for Large-scale Traffic Signals with Dense Feedback}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=UJTo1Gnkio},
}
Figure