Attention Gate in Traffic Forecasting

Anh Lam, Anh Nguyen, Bac Le

Abstract

Predicting urban mobility involves navigating complex spatio-temporal dynamics. This paper introduces an enhanced U-Net architecture for city-wide traffic forecasting, developed for the NeurIPS Traffic4Cast 2020 challenge. By integrating Attention Gates (AGs) into skip-connections, the model effectively filters spatial noise from non-traffic areas. Additionally, the architecture incorporates Dense Blocks for feature reuse and trigonometric time encoding to capture periodic temporal patterns. Experiments on datasets from Berlin, Istanbul, and Moscow demonstrate that the proposed model outperforms standard baselines in single-city forecasting, though challenges remain in multi-city generalization.

Introduction

Urban mobility prediction is a "wicked" problem characterized by intricate spatial and temporal dependencies. In the context of the NeurIPS Traffic4Cast 2020 challenge, the objective is to predict future traffic "frames"—essentially a movie of a city's traffic—based on historical sequences. Standard convolutional models often struggle with two primary issues: spatial noise and temporal context.

Spatial noise arises because grid-based city maps include many areas where traffic never occurs, such as parks, buildings, or bodies of water. Standard networks may waste computational resources on these irrelevant regions. Furthermore, traffic patterns are inherently periodic; a Monday morning differs significantly from a Sunday afternoon. Models that treat frames as isolated snapshots without understanding the underlying rhythm of the week often fail to capture these dynamics accurately.

Proposed Architecture

The authors build upon the U-Net architecture, a U-shaped network originally designed for medical image segmentation, which consists of a contraction path for context and an expansion path for localization. To adapt this for traffic forecasting, three key modifications were introduced.

Architecture of the proposed U-Net with Attention Gates, featuring Dense Blocks and skip-connection filtering.

Attention Gates (AGs)

The most significant modification is the replacement of simple skip-connections with Attention Gates. These gates utilize a "gating vector" from the lower stages of the network to filter features passed from the encoder to the decoder. By acting as a spatial mask, AGs highlight important road networks and prune away irrelevant background features before the reconstruction phase. This allows the model to focus its parameters on learning the dynamics of where traffic actually occurs.

Dense Blocks and Group Normalization

To improve feature reuse, standard convolution blocks were replaced with Dense Blocks. Additionally, the researchers swapped Batch Normalization for Group Normalization. This change is particularly beneficial when training with high-resolution spatial data, which often necessitates very small batch sizes (e.g., a batch size of 2) where Batch Normalization becomes unstable.

Trigonometric Time Encoding

To address the periodic nature of time, the model maps the time of day onto a trigonometric unit circle using sine and cosine functions. This representation ensures that the model understands the proximity between 11:55 PM and 12:05 AM, avoiding the artificial "jump" that occurs with simple linear indexing.

Experimental Setup

The study utilizes the Traffic4Cast 2020 dataset, covering Berlin, Istanbul, and Moscow. Each city is represented as a $495 \times 436$ grid, with each pixel corresponding to a $100m \times 100m$ area. Data is aggregated into 5-minute bins, resulting in 288 frames per day. Each frame contains nine channels (traffic volume, speed in four directions, and incident encoding) supplemented by seven channels of static map data, such as junction counts and points of interest.

Results and Discussion

The proposed model was evaluated against the 2020 challenge winner's baseline. In single-city evaluations, the Attention Gate model consistently achieved a lower Mean Squared Error (MSE) across all three cities.

The inclusion of extra temporal features (weekdays and time-of-day) provided a measurable performance boost. For instance, in Berlin, the MSE improved from 0.001415 to 0.001409 when these features were added. Furthermore, the experiments underscored the necessity of static map data; landmarks like hospitals and shops act as critical "sources" and "sinks" for traffic flow, and their inclusion is essential for precise forecasting.

The Generalization Paradox

An interesting finding emerged regarding multi-city generalization. While the model excelled when trained on a single city, its performance degraded when a single model was trained to handle all three cities simultaneously. This suggests that the attention mechanisms become highly specialized to a city's unique road layout. The "attention" learned for Berlin's specific geography does not translate effectively to Moscow's, highlighting a significant hurdle in developing universal traffic models.

Conclusion

By integrating Attention Gates and periodic time encoding, this research demonstrates that spatial constraints and temporal rhythms are fundamental to accurate traffic forecasting. While the model shows superior performance on individual cities, future work must address the challenge of multi-city generalization. Expanding the training set beyond the 20-day subset used in this study and incorporating external factors like weather or public events could further refine the model's predictive capabilities.

Summary updated

Abstract

Because of increased urban complexity and growing populations, more and more challenges about predicting city-wide mobility behavior are being organized. Traffic Map Movie Forecasting Challenge 2020 is secondly held in the competition track of the Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS). Similar to Traffic4Cast 2019, the task is to predict traffic flow volume, average speed in major directions on the geographical area of three big cities: Berlin, Istanbul, and Moscow. In this paper, we apply the attention mechanism on U-Net based model, especially we add an attention gate on the skip-connection between contraction path and expansion path. An attention gates filter features from the contraction path before combining with features on the expansion path, it enables our model to reduce the effect of non-traffic region features and focus more on crucial region features. In addition to the competition data, we also propose two extra features which often affect traffic flow, that are time and weekdays. We experiment with our model on the competition dataset and reproduce the winner solution in the same environment. Overall, our model archives better performance than recent methods.
BibTeX
@misc{lam2021attention,
  author = {Anh Lam and Anh Nguyen and Bac Le},
  title = {{Attention Gate in Traffic Forecasting}},
  year = {2021},
  eprint = {2109.13021},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.13021},
}
Figure