Dynamic Lane Traffic Signal Control with Group Attention and Multi-Timescale Reinforcement Learning

Qize Jiang, Jingze Li, Weiwei Sun, Baihua ZhengIJCAI 2021 · DOI

TL;DR

This paper introduces MT-GAD, a novel reinforcement learning approach that simultaneously controls traffic signals and dynamic (variable) lanes at intersections. By employing group attention for better generalization and a multi-timescale learning strategy to account for the differing temporal impacts of signal and lane changes, MT-GAD significantly outperforms existing methods in reducing traffic congestion.

What Problem Does This Paper Address?

Traffic congestion is a persistent urban problem, and traffic signals and lane management are key tools for mitigation. While intelligent traffic signal control using reinforcement learning (RL) has seen success, existing methods often overlook intersections with "dynamic lanes." These are lanes whose direction can be changed dynamically (e.g., to favor left turns during peak hours) to optimize traffic flow. The challenge lies in coordinating the control of traffic signals (which have immediate impacts) and dynamic lanes (which have more delayed effects) effectively. Current RL approaches for traffic signals often lack sophisticated structures to capture traffic data characteristics, leading to poor generalizability and slow convergence. Furthermore, controlling dynamic lanes is a relatively under-explored area, and integrating it with signal control presents a unique set of challenges due to their differing temporal dynamics.

Methods

The core of MT-GAD is a reinforcement learning framework that tackles the dual problem of traffic signal and dynamic lane control.

  • Traffic Flow Information Extraction: The system first extracts relevant information for each directional traffic flow (DTF) at an intersection. This includes vehicle counts, signal status, time since signal change, and predicted future traffic. Crucially, it separates inputs for traffic signal control (focusing on immediate impact) and dynamic lane control (considering longer-term effects).
  • Action Embedding Generation with Group Attention: To handle the large number of parameters and potential overfitting associated with standard attention mechanisms, MT-GAD introduces "group attention." This technique groups DTFs based on their impact on the current control phase (e.g., green vs. red signal) and applies attention within these groups. For dynamic lanes, it specifically considers the selected direction, the opposite direction, and a "coordinate" DTF that might share a green signal. This approach aims to improve generalizability and reduce computational complexity.
  • Multi-Timescale Training: Recognizing that traffic signals and dynamic lanes operate on different timescales, MT-GAD employs a multi-timescale learning strategy. This is achieved by using different discount factors ($\gamma$) for the RL agent when learning traffic signal control (a smaller $\gamma$ for short-term rewards) versus dynamic lane control (a larger $\gamma$ for longer-term rewards). The final action (a combination of signal phase and lane direction) is then derived from these two independently learned policies.

Datasets

The paper utilizes two newly developed datasets based on real taxi trajectories:

  • Porto (PT) dataset: Contains close to 1 million trajectories from July 2013 to March 2014.
  • Shanghai (SH) dataset: Contains over 11 million trajectories from April 2015.

These datasets were chosen because existing traffic signal control datasets are often too short (3600 seconds) to adequately evaluate the impact of dynamic lane changes, which can have longer-term effects. The authors simulated high-volume and low-volume traffic scenarios for these datasets to mimic real-world conditions.

Key Insights and Ideas

  • Interdependence of Signal and Lane Control: The paper highlights that traffic signal control and dynamic lane control are not independent problems but are interrelated and can benefit from joint optimization.
  • Temporal Mismatch is Key: The core insight is that the differing temporal dynamics (immediate vs. delayed impact) of traffic signals and dynamic lanes are critical and must be addressed explicitly in the learning process. This led to the multi-timescale training approach.
  • Group Attention for Efficiency and Generalization: Standard attention mechanisms can be parameter-heavy. Group attention offers a way to leverage attention's benefits while reducing parameters and improving generalization by exploiting structural properties of the traffic system.
  • Dynamic Lanes Offer Benefits, But Require Careful Control: The experiments show that dynamic lanes can improve traffic flow, but only when controlled effectively. Simple integration with existing RL methods doesn't always yield improvements, underscoring the need for specialized approaches like MT-GAD.

Key Results

  • Superior Performance: MT-GAD consistently outperforms traditional methods (FixedTime, SOTL) and other state-of-the-art RL approaches (PressLight, CoLight, FRAP) in reducing average vehicle delay (D) across various traffic volumes and datasets.
  • Significant Improvement with Dynamic Lanes: MT-GAD consistently outperforms its variant without dynamic lane control (MT-GAD w/o DL) at high traffic volumes, demonstrating the value of its integrated approach.
  • Enhanced Generalizability: MT-GAD shows superior generalizability, performing well when deployed on unseen intersections and even rotated traffic flow configurations, outperforming a variant using standard attention (MT-GAT*).
  • Effective Multi-Timescale Learning: The multi-timescale training approach is shown to be crucial. Variants using a single timescale (MT-GAD-0.8 or MT-GAD-0.95) performed worse, particularly in scenarios where dynamic lane changes were beneficial but required longer-term planning.
  • Robustness to Prediction Accuracy: The model is not overly sensitive to the accuracy of traffic flow prediction, as long as it reaches a minimum threshold.

Limitations and Future Work

  • Focus on Single Intersection: The current study focuses on a single intersection with one dynamic lane. The authors acknowledge that real-world traffic systems involve multiple interconnected intersections.
  • Future Work: The authors plan to extend their work to address control of traffic signals and reversible lanes, which represent another type of dynamic lane management. They also aim to explore multi-intersection coordination.

Summary updated

BibTeX
@inproceedings{jiang2021dynamic,
  author = {Qize Jiang and Jingze Li and Weiwei Sun and Baihua Zheng},
  title = {{Dynamic Lane Traffic Signal Control with Group Attention and Multi-Timescale Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/501},
}
Figure