Abstract
Spatiotemporal prediction is a critical task in fields ranging from meteorology to autonomous driving. While Convolutional LSTMs (ConvLSTMs) have long been the standard, their reliance on local convolutional operations limits their ability to capture global spatial dependencies. We present SwinLSTM, a novel recurrent architecture that integrates Swin Transformer blocks into the LSTM cell. By utilizing window-based self-attention, SwinLSTM achieves a global receptive field with linear computational complexity. Our experiments demonstrate that SwinLSTM significantly outperforms traditional CNN-based models across multiple benchmarks, including Moving MNIST and real-world traffic datasets.
Introduction
Predicting the future state of a system based on historical spatial and temporal data is a fundamental challenge in machine learning. For years, the ConvLSTM has been the dominant architecture, combining Long Short-Term Memory (LSTM) units for temporal modeling with Convolutional Neural Networks (CNNs) for spatial feature extraction. However, CNNs suffer from a "locality" problem: they process small pixel neighborhoods, requiring deep stacks of layers to build a sufficient receptive field for global dynamics. This often leads to computational inefficiency and a failure to capture long-range spatial correlations.
Methodology
The core innovation of this work is the SwinLSTM cell, which replaces standard convolutional layers with Swin Transformer blocks (STB).
The SwinLSTM Cell
Unlike ConvLSTMs that use separate convolutional weights for multiple gates, SwinLSTM simplifies the gating mechanism. It processes the input and the previous hidden state through a Swin Transformer Block and a Linear Projection (LP).
- Global Modeling: By employing Window-based Multi-head Self-Attention (W-MSA) and Shifted Window-based Multi-head Self-Attention (SW-MSA), the model computes relationships between distant regions of a frame efficiently.
- Simplified Gating: The architecture fuses traditional gates into a single "filter gate" ($F_t$) derived from the Transformer output, effectively managing the update and storage of information in the cell state.
Architecture Variants
We propose two primary configurations:
- SwinLSTM-B (Base): A streamlined single-cell architecture suitable for less complex spatiotemporal tasks.
- SwinLSTM-D (Deep): A multi-layer hierarchical architecture. It utilizes "Patch Merging" for downsampling and "Patch Expanding" for upsampling, enabling the model to learn multi-scale features and capture complex physical dynamics.
Experiments and Results
SwinLSTM was evaluated on four diverse benchmarks: Moving MNIST, Human3.6m (human action video), TaxiBJ (traffic flow), and KTH (human actions).
Quantitative Performance
SwinLSTM established new state-of-the-art (SOTA) benchmarks:
- Moving MNIST: Achieved a Mean Squared Error (MSE) of 17.7, a dramatic improvement over ConvLSTM (103.3) and the previous SOTA, CrevNet (22.3).
- KTH Dataset: In long-term prediction (40 frames), SwinLSTM reached a PSNR of 33.15, significantly outperforming STMFANet (27.56).
- TaxiBJ: Consistently yielded the lowest MSE across all time steps, proving its efficacy for real-world infrastructure forecasting.
Qualitative Analysis
Visual results highlight the model's ability to maintain structural integrity over time. While CNN-based models typically produce increasingly blurred predictions as the time horizon expands, SwinLSTM maintains sharp edges and accurate object trajectories.
Conclusion
SwinLSTM demonstrates that learning global spatial dependencies via self-attention is superior to the local inductive bias of convolutions for spatiotemporal tasks. By integrating the Swin Transformer into the recurrent loop, we achieve higher accuracy and better preservation of visual detail. Future work will focus on optimizing the computational footprint for real-time edge deployment and testing the architecture on more chaotic physical systems like fluid dynamics.
Summary updated