DynLight: Realize dynamic phase duration with multi-level traffic signal control

Liang Zhang, Shubin Xie, Jianming Deng

TL;DR

DynLight is a multi-level reinforcement learning (RL) framework for traffic signal control that decouples phase selection (which direction goes green) from duration selection (how long it stays green). By combining optimization-based phase picking with a Deep Q-Network for timing, it achieves state-of-the-art performance while offering a "cyclical" mode (DynLight-C) that is actually practical for real-world deployment.

What Problem Does This Paper Address?

Reinforcement learning has shown great promise in optimizing traffic lights, but existing models face three major hurdles:

  1. Rigidity: Most RL controllers use fixed time intervals (e.g., "pick a phase every 15 seconds"). This lacks the flexibility to handle the highly dynamic nature of real-world traffic.
  2. Unpredictability: Standard RL agents often pick phases in an arbitrary, non-cyclical order. While efficient for a computer, this is confusing and potentially dangerous for human drivers who expect a predictable sequence (e.g., North-South then East-West).
  3. Flawed Metrics: Common metrics like "average travel time" often ignore vehicles still stuck in the network when the simulation ends, leading to "fair-weather" results that don't reflect total system efficiency.

Methods

The authors propose DynLight, which breaks the control logic into two distinct levels:

  • Level 1: Phase Selection. This uses an optimization method called Max-QueueLength (M-QL). It looks at which lanes have the most cars waiting and selects the corresponding green light phase.
  • Level 2: Duration Selection. Once a phase is chosen, a Deep Q-Network (DQN) determines the optimal duration for that green light, choosing from a set of actions (e.g., 10 to 40 seconds in 5-second increments).

To bridge the gap between research and reality, they also introduce DynLight-C. This version replaces the M-QL phase selection with a fixed, cyclical order but keeps the trained DQN to dynamically adjust the timing of each phase in the cycle.

Finally, they introduce a more rigorous evaluation metric: Adjusted Average Travel Time, which extends the simulation clock until every single vehicle has cleared the network, ensuring no "hidden" congestion is left out of the data.

Datasets

The researchers evaluated their model using the CityFlow simulator on seven real-world traffic datasets from China:

  • JiNan Datasets: Three datasets representing a 3x4 grid of intersections.
  • HangZhou Datasets: Two datasets representing a 4x4 grid of intersections. These datasets include varied arrival rates (ranging from 0.83 to 1.94 vehicles per second) to test the model's robustness under different congestion levels.

Key Insights and Ideas

  • Decoupling is Key: By separating "what to do" from "how long to do it," the model gains the flexibility of traditional adaptive systems while retaining the learning power of RL.
  • Spatial Awareness Matters: The authors found that the best "state representation" wasn't just the total number of cars in a lane, but NV-S (Number of Vehicles on Segments). By dividing lanes into 100-meter chunks, the RL agent can "see" where the traffic jams are forming relative to the intersection.
  • The "Cyclical" Compromise: While DynLight (arbitrary phases) performs the best, DynLight-C (cyclical phases) still outperforms most existing RL baselines. This proves that we don't have to sacrifice driver sanity for efficiency; we just need smarter timing within the cycles.
  • Generalization: A model trained on one city's traffic patterns (e.g., JiNan) could be transferred to another (e.g., Hangzhou) and still maintain high performance, suggesting the "logic" of duration selection is somewhat universal.

Key Results

DynLight consistently outperformed traditional methods (FixedTime) and advanced RL baselines (like CoLight and MPLight):

  • Efficiency: In the JiNan1 dataset, DynLight achieved an adjusted average travel time of 255.16s, significantly lower than the next best RL method, AttentionLight (270.14s), and less than half the time of traditional FixedTime control (585.76s).
  • Deployment Readiness: DynLight-C (the cyclical version) remained highly competitive, often beating state-of-the-art non-cyclical RL models, making it a viable candidate for actual city infrastructure.

Limitations and Future Work

While DynLight represents a step forward, the authors note a few areas for growth:

  • Complexity: The current study focuses on standard 4-way intersections. Future work could explore more complex topologies, such as five-way intersections or those with dedicated U-turn lanes.
  • Advanced RL Techniques: The authors suggest that even more efficient state representations and newer RL architectures could further squeeze out performance gains.
  • Real-World Constraints: While DynLight-C addresses cyclical needs, other real-world constraints like pedestrian crossing times and emergency vehicle preemption were not explicitly modeled in this phase of the research.

Summary updated

Abstract

We would like to withdraw this article for the following reasons: 1 this article is not satisfactory for limited language and theoretical description; 2 we have enriched and revised this article with the help of other authors; 3 we must update the author contribution information.
BibTeX
@misc{zhang2022dynlight,
  author = {Liang Zhang and Shubin Xie and Jianming Deng},
  title = {{DynLight: Realize dynamic phase duration with multi-level traffic signal control}},
  year = {2022},
  eprint = {2204.03471},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.03471},
}
Figure