Time Difference Penalized Traffic Signal Timing by LSTM Q-Network to Balance Safety and Capacity at Intersections

Lyuchao Liao, Jierui Liu, Xinke Wu, Fumin Zou, Jeng-Shyang Pan, Qi Sun, Shengbo Eben Li, Maolin ZhangIEEE Access 2020 · DOI

Abstract

Urban traffic signal control faces a persistent challenge in balancing intersection throughput with driver safety and compliance. Traditional models often prioritize vehicle flow at the expense of excessive wait times on minor lanes, leading to "red-light overflow" and potential traffic violations. This paper presents Rep-DRQN, a Time Difference Penalized LSTM Q-Network designed to optimize signal timing by incorporating driver patience into the reward mechanism. By utilizing an LSTM-based architecture to capture temporal vehicle dynamics and a novel reward function that penalizes red-light durations exceeding a bearable threshold, the proposed system improves efficiency by up to 14% while significantly reducing safety risks associated with driver frustration.

Introduction

The management of urban traffic is a complex optimization problem where capacity and safety are often at odds. While adaptive signal timing using artificial intelligence has shown promise in maximizing throughput, many existing models overlook the human element of driving. Specifically, if an algorithm indefinitely prioritizes high-volume lanes, drivers in lower-volume lanes may face intolerable wait times. This phenomenon, known as red-light overflow, frequently results in illegal red-light running as driver patience is exhausted. Furthermore, standard models often simplify traffic states into static queue lengths, failing to account for the nuanced dynamics of lane-changing and car-following that influence real-world intersection performance.

Methodology: The Rep-DRQN Framework

To address these challenges, we propose the Time Difference Penalized LSTM Q-Network (Rep-DRQN). This framework integrates deep reinforcement learning with a focus on both microscopic traffic behavior and temporal dependencies.

Overview of the Rep-DRQN reinforcement learning framework for traffic signal control.

Microcosmic State Representation

Unlike models that rely on simple vehicle counts, Rep-DRQN utilizes a high-fidelity state representation. The environment is encoded into an 85-bit vector that captures vehicle positions, steering lamp status (to anticipate lane changes), current signal phases, and the elapsed duration of red lights. This allows the agent to perceive "turbulence" in traffic flow before it manifests as a full-scale slowdown.

Conversion of microscopic traffic states into a structured state matrix.

LSTM Architecture and Reward Function

Standard Deep Q-Networks (DQN) treat traffic states as independent snapshots. In contrast, our model employs Long Short-Term Memory (LSTM) cells to maintain a temporal "memory" of traffic patterns. This is crucial for understanding the evolving nature of vehicle interactions over time.

The core innovation lies in the Behavior-Aware Reward Function. The reward signal incorporates a penalty for any lane exceeding a Bearable Red-light Duration (BRD), set at 300 seconds. By applying a square-root penalty to excessive wait times, the model is incentivized to transition signals smoothly, preventing "panicked" switching while ensuring that no driver is left waiting indefinitely.

Experimental Evaluation

The Rep-DRQN model was evaluated using the SUMO (Simulation of Urban MObility) package, simulating a standard four-way intersection with 12 entry lanes. To accelerate training, 12 simultaneous simulation environments were used for parallel data sampling.

Key Results

The proposed model was compared against Fixed Signal Timing (FST) and standard DQN baselines. Key findings include:

  • Efficiency Gains: Rep-DRQN improved intersection throughput by 14.28% over fixed timing and 5.26% over standard DQN.
  • Reduced Congestion: The average number of waiting vehicles was reduced to approximately 40, a significant improvement over the ~65 vehicles observed with fixed timing.
  • Enhanced Travel Time: Vehicles traversed the intersection roughly 45 seconds faster than with fixed timing.
  • Safety and Compliance: The model successfully eliminated red-light overflow by keeping all wait times below the 300-second threshold, thereby reducing the likelihood of traffic violations.

Conclusion and Future Work

Rep-DRQN demonstrates that integrating driver psychology—specifically the limit of patience—into reinforcement learning rewards can create traffic systems that are both efficient and safe. By moving beyond simple throughput metrics, the model ensures a more equitable and predictable flow of traffic. Future research will focus on extending this approach to multi-intersection coordination, enabling the optimization of entire urban traffic corridors through collaborative reinforcement learning.

Summary updated

BibTeX
@article{liao2020time,
  author = {Lyuchao Liao and Jierui Liu and Xinke Wu and Fumin Zou and Jeng-Shyang Pan and Qi Sun and Shengbo Eben Li and Maolin Zhang},
  title = {{Time Difference Penalized Traffic Signal Timing by LSTM Q-Network to Balance Safety and Capacity at Intersections}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.2989151},
}
Figure