Abstract
Traffic Signal Control (TSC) aims to reduce the average travel time of vehicles in a road network, which in turn enhances fuel utilization efficiency, air quality, and road safety, benefiting society as a whole. Due to the complexity of long-horizon control and coordination, most prior TSC methods leverage deep reinforcement learning (RL) to search for a control policy and have witnessed great success. However, TSC still faces two significant challenges. 1) The travel time of a vehicle is delayed feedback on the effectiveness of TSC policy at each traffic intersection since it is obtained after the vehicle has left the road network. Although several heuristic reward functions have been proposed as substitutes for travel time, they are usually biased and not leading the policy to improve in the correct direction. 2) The traffic condition of each intersection is influenced by the non-local intersections since vehicles traverse multiple intersections over time. Therefore, the TSC agent is required to leverage both the local observation and the non-local traffic conditions to predict the long-horizontal traffic conditions of each intersection comprehensively. To address these challenges, we propose DenseLight, a novel RL-based TSC method that employs an unbiased reward function to provide dense feedback on policy effectiveness and a non-local enhanced TSC agent to better predict future traffic conditions for more precise traffic control. Extensive experiments and ablation studies demonstrate that DenseLight can consistently outperform advanced baselines on various road networks with diverse traffic flows.
Introduction
Traffic Signal Control (TSC) is a fundamental component of urban infrastructure management, directly impacting travel time, fuel consumption, and air quality. While Deep Reinforcement Learning (RL) has emerged as a powerful tool for optimizing traffic flow, it faces two primary bottlenecks: delayed feedback and limited coordination.
The ultimate metric for TSC success is the minimization of average travel time. However, in an RL environment, this metric is inherently delayed; a vehicle's total travel time is only known once it exits the network, long after specific control actions were taken. Furthermore, traffic is a global, interconnected system where the state of one intersection is influenced by distant traffic flows. Most current models rely on local or near-neighbor communication, failing to capture the "big picture" of urban mobility. DenseLight addresses these issues through a mathematically unbiased reward function and a non-local communication architecture.
The DenseLight Framework
1. Ideal-Factual Distance Gap (IFDG)
To solve the problem of delayed feedback, researchers often use heuristic rewards like queue length or "pressure" (the difference between incoming and outgoing vehicles). However, these heuristics are often biased and may not align perfectly with the goal of minimizing total travel time.
DenseLight introduces the Ideal-Factual Distance Gap (IFDG). This reward function calculates the difference between the distance a vehicle could have traveled at maximum speed and the distance it actually traveled during a single time step. The authors provide a mathematical proof that minimizing this gap over time is equivalent to minimizing total travel time. This provides the RL agent with dense, unbiased feedback at every second, allowing for more precise policy updates.
2. Non-local Enhanced TSC (NL-TSC) Agent
Traffic conditions at a specific intersection are often the result of vehicle movements several blocks away. DenseLight utilizes a Non-local enhanced TSC (NL-TSC) agent that incorporates Spatial-Temporal Augmented Observations. This architecture allows the agent to consider its 2D position and previous states to better understand evolving traffic trends.
3. Dense Communication Layers (DCL)
Standard coordination methods typically limit communication to immediate 1-hop or 2-hop neighbors. DenseLight breaks this constraint using Dense Communication Layers (DCL). By employing a learnable communication matrix, the model can automatically identify which distant intersections are most relevant to local timing decisions. To maintain computational efficiency as the network scales, the authors use a bottleneck approach, decomposing large weight matrices into smaller, more manageable components.
Experimental Evaluation
The performance of DenseLight was evaluated using the CityFlow simulator across four real-world road networks and one synthetic dataset:
- Jinan12 & Hangzhou16: Standard urban grids in China.
- NewYork48 & NewYork196: Large-scale grids in the USA, with the latter representing a massive 28x7 network.
- Synthetic Fluctuating Flow ($D_{JN12(f)}$): A dataset designed to test robustness against sudden surges in traffic volume.
DenseLight was compared against traditional methods like MaxPressure and state-of-the-art RL baselines including CoLight and MPLight.
Key Results and Insights
Superior Performance at Scale
DenseLight consistently outperformed all baselines. On the largest network (NewYork196), it reduced average travel time to 803 seconds, a 20% improvement over the previous best-performing model (~1004 seconds). This demonstrates the framework's ability to handle the complexities of large-scale urban environments.
Robustness to Traffic Fluctuations
In tests using the fluctuating flow dataset, DenseLight maintained stable performance while other models struggled to adapt to changing arrival rates. This suggests that the IFDG reward and non-local communication provide a more resilient understanding of traffic dynamics than traditional heuristics.
The Importance of Non-local Communication
Weight visualization within the DCL revealed a critical insight: the most influential intersection for a controller is not always its immediate neighbor. By allowing for non-local communication, DenseLight successfully captures the complex ways traffic waves propagate across a city, leading to more accurate value predictions and better control decisions.
Conclusion
DenseLight represents a significant advancement in Traffic Signal Control by providing a solution to the dual challenges of biased feedback and local-only coordination. By utilizing the unbiased IFDG reward and a non-local communication architecture, it achieves state-of-the-art results on large-scale networks. Future research will focus on improving the model's generalization across different cities and its resilience to noisy or incomplete sensor data.
Summary updated