AttendLight: Universal Attention-Based Reinforcement Learning Model for Traffic Signal Control

Afshin Oroojlooy, MohammadReza Nazari, Davood Hajinezhad, Jorge Silva

TL;DR

AttendLight is a "universal" reinforcement learning (RL) model for traffic signal control that can be deployed at any intersection without retraining. By utilizing a dual-attention architecture, it seamlessly handles varying numbers of lanes and signal phases, outperforming both traditional rule-based systems and state-of-the-art specialized RL models.

The Scalability Bottleneck in Traffic Control

Traffic signal control (TSC) is a classic optimization problem, but traditional Reinforcement Learning approaches suffer from a significant scalability bottleneck: lack of generalization.

Most existing RL models are "one-offs"—they are trained for a specific intersection topology, such as a standard 4-way crossroad. If you attempt to deploy that model to a 3-way "T" intersection or a road with an extra turn lane, the model's input and output dimensions break. This requires a complete redesign and retraining from scratch. In a city with thousands of unique intersections, this "train-per-intersection" requirement is computationally expensive and practically unfeasible.

To solve this, the authors propose AttendLight, an end-to-end RL algorithm designed to be a single, universal model capable of controlling any intersection configuration it might encounter.

Methods: A Dual-Attention Architecture

The core innovation of AttendLight is the use of two distinct attention mechanisms that decouple the model's architecture from the physical structure of the intersection.

  1. State Attention: This layer handles the variable number of approaching lanes. It takes raw traffic characteristics (like the number of moving and waiting vehicles) and embeds them into a higher-dimensional space. The attention mechanism then aggregates these lane-level features into a fixed-length "phase representation," regardless of how many lanes are involved in that phase.
  2. Action Attention: This layer handles the variable number of possible signal phases (actions). It compares the current "phase representations" against the output of a Long Short-Term Memory (LSTM) network, which tracks the history of previous signal changes. This allows the model to decide which phase to activate next, even if the total number of available phases differs between intersections.
The AttendLight architecture uses dual attention to handle variable inputs (lanes) and outputs (phases).

The model is trained using a variance-reduced version of the REINFORCE algorithm. The reward function is based on minimizing "intersection pressure"—the difference between the number of vehicles waiting in entering lanes and those in leaving lanes—which serves as a proven proxy for reducing overall travel time.

Datasets and Topologies

The researchers evaluated AttendLight using a mix of real-world and synthetic data across 112 unique intersection instances:

  • Real-World Data: Traffic flow datasets from Hangzhou and Atlanta, which are commonly used as benchmarks in TSC research.
  • Synthetic Data: Created to cover scenarios lacking real-world data, specifically intersections with three lanes per road.
  • Topologies: The experiments covered 11 different intersection topologies, ranging from 3-way to 4-way structures with varying lane counts and signal phase options (from 2 to 8 phases).
An example of the complex 3-way intersection topologies handled by the model.

Key Insights and Ideas

  • Attention as a Structural Bridge: While attention is famous for its role in Natural Language Processing, this paper demonstrates its power as a structural tool. It allows the model to treat an intersection as a flexible graph rather than a rigid matrix, enabling the "universal" property.
  • Knowledge Sharing: By training on a diverse set of intersections (the "multi-env" regime), the model learns fundamental principles of traffic flow. The authors found that a model trained on many intersections can actually perform better on some specific cases than a model trained only on that single case, likely due to "knowledge sharing" of successful control strategies across similar patterns.
  • Zero-Shot Generalization: The model demonstrates impressive performance on "unseen" intersections. This suggests that the attention mechanisms successfully extract abstract representations of traffic status that are invariant to the specific geometry of the road.

Key Results

AttendLight was tested in two regimes: single-environment (trained and tested on one intersection) and multi-environment (one model for all).

  • Superiority over Baselines: In the single-env regime, AttendLight outperformed classical methods (like MaxPressure and SOTL) and state-of-the-art RL models (like FRAP). It achieved a 46% improvement over Fixed-Time policies and a 9% improvement over FRAP.
  • Robust Universality: In the multi-env regime, a single AttendLight model achieved performance very close to specialized models. On average, there was only a 15% performance gap when using one universal model for 112 different intersections compared to training 112 separate models.
  • Fast Calibration: For critical intersections where maximum performance is needed, the authors showed that a universal model could be "fine-tuned" (Few-Shot training) to reach near-optimal performance in just 200-1,000 episodes, compared to the 100,000+ episodes usually required to train from scratch.

Limitations and Future Work

  • Isolated Intersections: The current study focuses on controlling single intersections in isolation. In the real world, intersections are linked in a network. Extending AttendLight to coordinated multi-intersection control is a vital next step.
  • Extreme Topologies: While the model is flexible, it may still struggle with extremely unusual or complex intersection designs that are significantly different from anything in the training set.
  • Algorithm Improvements: The authors used the REINFORCE algorithm for its simplicity in demonstrating the architecture's effectiveness, but they suggest that more advanced RL algorithms like Actor-Critic or PPO could further improve numerical results.

Summary updated

Abstract

We propose AttendLight, an end-to-end Reinforcement Learning (RL) algorithm for the problem of traffic signal control. Previous approaches for this problem have the shortcoming that they require training for each new intersection with a different structure or traffic flow distribution. AttendLight solves this issue by training a single, universal model for intersections with any number of roads, lanes, phases (possible signals), and traffic flow. To this end, we propose a deep RL model which incorporates two attention models. The first attention model is introduced to handle different numbers of roads-lanes; and the second attention model is intended for enabling decision-making with any number of phases in an intersection. As a result, our proposed model works for any intersection configuration, as long as a similar configuration is represented in the training set. Experiments were conducted with both synthetic and real-world standard benchmark datasets. Our numerical experiment covers intersections with three or four approaching roads; one-directional/bi-directional roads with one, two, and three lanes; different number of phases; and different traffic flows. We consider two regimes: (i) single-environment training, single-deployment, and (ii) multi-environment training, multi-deployment. AttendLight outperforms both classical and other RL-based approaches on all cases in both regimes.
BibTeX
@inproceedings{oroojlooy2020attendlight,
  author = {Afshin Oroojlooy and MohammadReza Nazari and Davood Hajinezhad and Jorge Silva},
  title = {{AttendLight: Universal Attention-Based Reinforcement Learning Model for Traffic Signal Control}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=SlsdzrVxGeu},
}
Figure