Sequence Decision Transformer for Adaptive Traffic Signal Control

Rui Zhao, Haofeng Hu, Yun Li, Yuze Fan, Fei Gao, Zhenhai Gao

TL;DR

This paper introduces the Sequence Decision Transformer (SDT), a novel deep reinforcement learning approach for adaptive traffic signal control (ATSC). By leveraging transformer architectures, similar to those used in natural language processing, SDT effectively captures complex traffic dynamics, leading to significant improvements in traffic flow efficiency and congestion reduction compared to existing methods.

What Problem Does This Paper Address?

Urban traffic congestion is a major global issue, leading to economic losses and environmental degradation. Traditional traffic signal systems often rely on fixed timings, failing to adapt to real-time traffic conditions. While Adaptive Traffic Signal Control (ATSC) aims to address this, current deep reinforcement learning (DRL) approaches struggle with the high dimensionality and dynamic nature of traffic data, leading to issues like non-stationarity during training and poor generalization to new environments. This paper tackles the challenge of developing a more robust and efficient ATSC system that can handle complex traffic scenarios and learn optimal policies with greater speed and accuracy.

Methods

The core innovation is the Sequence Decision Transformer (SDT), a DRL agent built upon an encoder-decoder transformer architecture.

  • Problem Formulation: The ATSC problem is framed as a Markov Decision Process (MDP), defining specific observation spaces (vehicle counts, speeds, queue lengths, signal phases), action spaces (selecting traffic signal phases), and a multi-objective reward function that balances vehicle count, average speed, queue length, and waiting time.
  • SDT Architecture:
    • Encoder: Processes traffic observations, learns latent representations, and estimates the value function. It uses self-attention mechanisms to capture dependencies across the input sequence.
  • Decoder: Acts as the policy network, taking the encoder's output to generate the optimal traffic signal action.
  • Training: The model is trained using Proximal Policy Optimization (PPO), a robust reinforcement learning algorithm known for its stability and sample efficiency. PPO is used to update the policy network, ensuring that policy updates remain within a controlled range to prevent destabilizing the learning process.
  • Key Components: The SDT leverages transformer's self-attention mechanism for parallel processing and capturing long-range dependencies, which is crucial for understanding complex traffic patterns. It also incorporates an actor-critic structure where the encoder acts as the critic (value estimation) and the decoder as the actor (policy generation).

Datasets

The experiments were conducted using the Simulation of Urban Mobility (SUMO) traffic simulation tool. The paper describes three synthetic traffic scenarios:

  • Easy Scenario: A simple intersection with two lanes per direction, traffic density of 600 veh/h.
  • Medium Scenario: A more complex intersection with four lanes per direction, traffic density of 4000 veh/h.
  • Hard Scenario: A highly complex intersection with four straight, two right-turn, and two left-turn lanes per direction, traffic density of 5000 veh/h.

These scenarios were designed to simulate realistic traffic conditions and test the algorithm's performance under varying levels of complexity.

Key Insights and Ideas

  • Transformer for Traffic: Applying transformer architectures, successful in NLP, to traffic signal control is a promising direction. The self-attention mechanism is particularly well-suited for capturing the complex, sequential, and interdependent nature of traffic flow.
  • Sequence Decision Modeling: Framing ATSC as a sequence decision problem allows the model to learn policies that consider historical traffic patterns and predict future states more effectively.
  • Multi-Objective Reward Balancing: The use of a weighted reward function is critical for balancing competing objectives (e.g., minimizing vehicle count vs. maximizing speed) and preventing the agent from over-optimizing for a single metric, which could lead to poor overall performance.
  • PPO for Stability: Integrating PPO with the transformer architecture provides a stable and efficient learning mechanism, crucial for handling the non-stationarity inherent in traffic environments.

Key Results

The SDT model demonstrated significant improvements over baseline methods across various metrics and scenarios:

  • Overall Performance: SDT showed improvements of 26.8%, 150%, and 21.7% over traditional ATSC algorithms (FIXED, PPO, DQN) and 18%, 30%, and 15.6% over the state-of-the-art FRAP algorithm in terms of vehicle number, average speed, and queue length, respectively, particularly in the harder scenarios.
  • Vehicle Number: SDT consistently reduced the total number of vehicles at intersections, indicating better traffic flow management.
  • Average Speed: The model significantly increased average vehicle speeds, suggesting reduced congestion and faster travel times.
  • Queue Length: SDT effectively reduced queue lengths, leading to smoother traffic flow and less waiting.
  • Training Efficiency: The paper notes that the SDT approach can significantly reduce training times and manage larger observation spaces more effectively compared to traditional DRL methods.

Limitations and Future Work

The authors acknowledge the following limitations and suggest future research directions:

  • Computational Resources: The SDT algorithm is computationally intensive, requiring significant resources for training. Future work aims to optimize computational efficiency.
  • Hyperparameter Sensitivity: The algorithm's performance can be sensitive to hyperparameter settings, necessitating careful tuning. Research will focus on automated hyperparameter tuning techniques to enhance robustness.
  • Scalability: Reducing computational demands is key to broadening SDT's applicability and scalability to larger, more complex urban networks.
  • Real-World Deployment: While tested in simulation, further research is needed to bridge the gap between simulated performance and real-world deployment, addressing potential challenges in sensor data integration and system reliability.

Summary updated

Abstract

Urban traffic congestion poses significant economic and environmental challenges worldwide. To mitigate these issues, Adaptive Traffic Signal Control (ATSC) has emerged as a promising solution. Recent advancements in deep reinforcement learning (DRL) have further enhanced ATSC’s capabilities. This paper introduces a novel DRL-based ATSC approach named the Sequence Decision Transformer (SDT), employing DRL enhanced with attention mechanisms and leveraging the robust capabilities of sequence decision models, akin to those used in advanced natural language processing, adapted here to tackle the complexities of urban traffic management. Firstly, the ATSC problem is modeled as a Markov Decision Process (MDP), with the observation space, action space, and reward function carefully defined. Subsequently, we propose SDT, specifically tailored to solve the MDP problem. The SDT model uses a transformer-based architecture with an encoder and decoder in an actor–critic structure. The encoder processes observations and outputs, both encoded data for the decoder, and value estimates for parameter updates. The decoder, as the policy network, outputs the agent’s actions. Proximal Policy Optimization (PPO) is used to update the policy network based on historical data, enhancing decision-making in ATSC. This approach significantly reduces training times, effectively manages larger observation spaces, captures dynamic changes in traffic conditions more accurately, and enhances traffic throughput. Finally, the SDT model is trained and evaluated in synthetic scenarios by comparing the number of vehicles, average speed, and queue length against three baselines, including PPO, a DQN tailored for ATSC, and FRAP, a state-of-the-art ATSC algorithm. SDT shows improvements of 26.8%, 150%, and 21.7% over traditional ATSC algorithms, and 18%, 30%, and 15.6% over the FRAP. This research underscores the potential of integrating Large Language Models (LLMs) with DRL for traffic management, offering a promising solution to urban congestion.
BibTeX
@inproceedings{zhao2024sequence,
  author = {Rui Zhao and Haofeng Hu and Yun Li and Yuze Fan and Fei Gao and Zhenhai Gao},
  title = {{Sequence Decision Transformer for Adaptive Traffic Signal Control}},
  booktitle = {Sensors},
  year = {2024},
  url = {https://openreview.net/forum?id=RJJMEGBn9J},
}
Figure