Introduction
Traffic forecasting has transitioned from simple time-series analysis into a complex spatiotemporal "video prediction" task. Modern datasets represent urban traffic as a sequence of image-like frames where each pixel encodes critical information such as vehicle volume and speed. While Convolutional Neural Networks (CNNs) and U-Net architectures have traditionally dominated this field, they often struggle to capture long-range dependencies and the intricate inter-relationships between disparate time steps and data channels.
Transformers offer a solution for modeling these long-range interactions but are typically computationally prohibitive for high-resolution 3D data. SwinUNet3D addresses this by integrating the efficiency of Video Swin Transformers into a hierarchical U-Net framework, creating a model that is both computationally viable and highly effective at capturing 3D traffic dynamics.
Methodology
SwinUNet3D is a pure transformer-based hierarchical architecture that eschews traditional convolutional blocks in its primary stages. The model's design is centered around several key innovations:
3D Shifted Window Transformers
To avoid the quadratic computational cost of global attention, SwinUNet3D utilizes "shifted windows." Attention is computed within small, local 3D blocks. In subsequent layers, these windows are shifted, allowing information to propagate and mix across the entire spatiotemporal field.
Feature Mixing Layer
A critical component of the architecture is the specialized "feature mixing" layer. Before entering the transformer blocks, the model flattens the time and channel dimensions and processes them through a fully connected layer. This allows the network to learn the inter-relationships between features—such as the correlation between speed and volume—across time before spatial processing begins.
Asymmetric U-Net Structure
To manage the high parameter count inherent in transformer models, the authors implemented an asymmetric structure. The encoder is "heavy," featuring four transformer layers per block, while the decoder is "light," utilizing only one transformer layer per block. This design ensures the model remains trainable on standard hardware.
Experimental Setup and Datasets
The model was evaluated using the Traffic Map Movie Forecasting Challenge 2021 (Traffic4cast2021) dataset. This benchmark includes traffic data from 10 diverse cities over a two-year period. The task involves taking 12 frames of traffic state (representing one hour) and predicting the subsequent 6 frames. The input features consist of 8 dynamic channels (volume and speed for four cardinal directions) and 9 static channels (road map geometry).
Results and Discussion
SwinUNet3D demonstrates superior performance compared to standard graph-based and convolution-based baselines provided by the Traffic4cast competition.
The results indicate that the transformer-based approach is better suited for capturing the complex dependencies in traffic data. Ablation studies further highlighted the importance of the Feature Mixing Layer; its inclusion reduced the Core MSE from 49.99 to 49.72, suggesting that understanding feature interactions over time is as vital as spatial localization.
Conclusion
SwinUNet3D proves that a U-Net architecture can function effectively for 3D spatiotemporal tasks without relying on traditional convolutional blocks. While the model is more resource-intensive than 2D CNNs and requires substantial data, its ability to model complex traffic patterns sets a new benchmark. Future research will focus on multi-objective loss formulations and hypercomplex networks to further refine token mixing and predictive accuracy.
Summary updated