TL;DR
This paper introduces CoordLight, a novel framework for decentralized traffic signal control that significantly reduces average travel times in urban networks. It achieves this by introducing a new "Queue Dynamic State Encoding" (QDSE) for richer traffic state representation and a "Neighbor-aware Policy Optimization" (NAPO) algorithm that enables more effective coordination between traffic signals.
What Problem Does This Paper Address?
Urbanization leads to increasing traffic congestion, which negatively impacts travel times, pollution, and quality of life. Adaptive Traffic Signal Control (ATSC) aims to alleviate this by dynamically adjusting signal timings. While Multi-Agent Reinforcement Learning (MARL) has shown promise, existing decentralized MARL approaches struggle with two main challenges:
- Partial Observability and State Representation: Individual traffic signals (agents) only have local information, making it hard to understand the broader network state and predict future congestion. Current state representations often lack the richness to capture complex traffic dynamics.
- Decentralized Coordination: Achieving effective coordination among many decentralized agents is difficult. Agents can become myopic, focusing only on their immediate surroundings, which hinders efficient network-wide optimization. Existing coordination methods often struggle with scalability and training stability.
Methods
CoordLight tackles these challenges with two core innovations:
- Queue Dynamic State Encoding (QDSE): This novel state representation goes beyond simple vehicle counts or pressure. It incorporates features derived from vehicle queueing dynamics, including:
- Current queue length.
- Entering and leaving vehicle counts.
- Information about moving vehicles approaching the intersection.
-
Distance to the foremost moving vehicle. This richer representation allows agents to better predict upcoming congestion and understand local traffic dynamics.
-
Neighbor-aware Policy Optimization (NAPO): This is a fully decentralized MARL algorithm that builds upon Independent Proximal Policy Optimization (IPPO). Key features include:
- Attention-based Actor-Critic Network: An augmented actor-critic architecture uses multi-head attention to dynamically weigh the importance of neighboring agents' states and actions. This allows agents to focus on influential neighbors and better understand their impact.
- Neighbor-aware Advantage Calculation: The advantage function is modified to incorporate the weighted influence of neighbors, leading to more accurate estimates of an agent's contribution to regional objectives and a more stable training process.
- Auxiliary Prediction Losses: The model also includes auxiliary losses for predicting future queue lengths, further enhancing its predictive capabilities.
Datasets
The experiments were conducted using real-world traffic road networks and traffic demand datasets from the CityFlow simulator. Specifically, the paper used:
- Jinan, China: A 3x4 intersection network.
- Hangzhou, China: A 4x4 intersection network.
- New York, USA: A large-scale 7x28 intersection network (196 intersections).
These datasets were used with different traffic demand levels (e.g., DJN(1), DJN(2), DJN(3) for Jinan) to evaluate the robustness of the proposed method.
Key Insights and Ideas
- Richer State Representation is Crucial: Simply counting vehicles or measuring pressure isn't enough. Understanding queueing dynamics and predicting future vehicle arrivals significantly improves an agent's ability to make proactive, rather than reactive, decisions.
- Targeted Coordination Outperforms Broad Cooperation: By using attention mechanisms to identify and weigh the influence of specific neighbors, NAPO can achieve more effective coordination than methods that treat all neighbors equally or rely on more general cooperation signals.
- Balancing Complexity and Accuracy: QDSE strikes a good balance between providing rich traffic information and maintaining manageable complexity, making it practical for real-world deployment.
- Robustness to Sensor Noise: The QDSE state representation demonstrated good robustness to simulated sensor noise, suggesting its potential for real-world application where perfect sensor data is rare.
Key Results
CoordLight consistently outperformed existing state-of-the-art methods across diverse traffic networks and demand scenarios.
- Average Travel Time Reduction: CoordLight achieved the lowest average travel times. For example, on the challenging New York dataset (196 intersections), it significantly reduced travel times compared to leading baselines. On the DJN(1) dataset, it achieved an average travel time of 61.58s, outperforming SocialLight by a notable margin.
- Improved Coordination: The paper showed that CoordLight consistently achieved the lowest average travel time and variance across intersections, indicating superior coordination performance.
- Ablation Studies:
- QDSE significantly improved performance over simpler state representations like Vehicle Counts, General Pressure, Efficient Pressure, and Advanced Traffic State.
- Removing the spatio-temporal network (STN) led to rapid convergence to sub-optimal solutions, highlighting the importance of capturing spatial and temporal dependencies.
- The state-action decoder was crucial for capturing dependencies among agents and stabilizing training.
Limitations and Future Work
- Homogeneous Policy: The current implementation trains a homogeneous policy, meaning all agents share the same policy weights. Future work could explore heterogeneous policies for more diverse intersection types.
- Simulation-Based: Experiments were conducted in simulators. While robustness to sensor noise was tested, real-world deployment would require further validation.
- Future Directions: The authors suggest extending the method to:
- Heterogeneous networks with diverse intersection layouts and lane configurations.
- More realistic, asynchronous signal control settings.
- Handling imperfect sensor data, traffic priorities (e.g., emergency vehicles), and scenarios like traffic accidents and road closures.
Summary updated