SwinLSTM: Improving Spatiotemporal Prediction Accuracy using Swin Transformer and LSTM

Song Tang, Chuang Li, Pu Zhang, Rongnian TangICCV 2023 · DOI · arXiv

Abstract

Spatiotemporal prediction is a critical task in fields ranging from meteorology to autonomous driving. While Convolutional LSTMs (ConvLSTMs) have long been the standard, their reliance on local convolutional operations limits their ability to capture global spatial dependencies. We present SwinLSTM, a novel recurrent architecture that integrates Swin Transformer blocks into the LSTM cell. By utilizing window-based self-attention, SwinLSTM achieves a global receptive field with linear computational complexity. Our experiments demonstrate that SwinLSTM significantly outperforms traditional CNN-based models across multiple benchmarks, including Moving MNIST and real-world traffic datasets.

Introduction

Predicting the future state of a system based on historical spatial and temporal data is a fundamental challenge in machine learning. For years, the ConvLSTM has been the dominant architecture, combining Long Short-Term Memory (LSTM) units for temporal modeling with Convolutional Neural Networks (CNNs) for spatial feature extraction. However, CNNs suffer from a "locality" problem: they process small pixel neighborhoods, requiring deep stacks of layers to build a sufficient receptive field for global dynamics. This often leads to computational inefficiency and a failure to capture long-range spatial correlations.

Comparison of transition operations in FC-LSTM, ConvLSTM, and the proposed SwinLSTM.

Methodology

The core innovation of this work is the SwinLSTM cell, which replaces standard convolutional layers with Swin Transformer blocks (STB).

The SwinLSTM Cell

Unlike ConvLSTMs that use separate convolutional weights for multiple gates, SwinLSTM simplifies the gating mechanism. It processes the input and the previous hidden state through a Swin Transformer Block and a Linear Projection (LP).

  • Global Modeling: By employing Window-based Multi-head Self-Attention (W-MSA) and Shifted Window-based Multi-head Self-Attention (SW-MSA), the model computes relationships between distant regions of a frame efficiently.
  • Simplified Gating: The architecture fuses traditional gates into a single "filter gate" ($F_t$) derived from the Transformer output, effectively managing the update and storage of information in the cell state.

Architecture Variants

We propose two primary configurations:

  1. SwinLSTM-B (Base): A streamlined single-cell architecture suitable for less complex spatiotemporal tasks.
  2. SwinLSTM-D (Deep): A multi-layer hierarchical architecture. It utilizes "Patch Merging" for downsampling and "Patch Expanding" for upsampling, enabling the model to learn multi-scale features and capture complex physical dynamics.

Experiments and Results

SwinLSTM was evaluated on four diverse benchmarks: Moving MNIST, Human3.6m (human action video), TaxiBJ (traffic flow), and KTH (human actions).

Quantitative Performance

SwinLSTM established new state-of-the-art (SOTA) benchmarks:

  • Moving MNIST: Achieved a Mean Squared Error (MSE) of 17.7, a dramatic improvement over ConvLSTM (103.3) and the previous SOTA, CrevNet (22.3).
  • KTH Dataset: In long-term prediction (40 frames), SwinLSTM reached a PSNR of 33.15, significantly outperforming STMFANet (27.56).
  • TaxiBJ: Consistently yielded the lowest MSE across all time steps, proving its efficacy for real-world infrastructure forecasting.

Qualitative Analysis

Visual results highlight the model's ability to maintain structural integrity over time. While CNN-based models typically produce increasingly blurred predictions as the time horizon expands, SwinLSTM maintains sharp edges and accurate object trajectories.

Qualitative comparison of video prediction on Moving MNIST. SwinLSTM maintains sharp edges compared to the blurry results of ConvLSTM.

Conclusion

SwinLSTM demonstrates that learning global spatial dependencies via self-attention is superior to the local inductive bias of convolutions for spatiotemporal tasks. By integrating the Swin Transformer into the recurrent loop, we achieve higher accuracy and better preservation of visual detail. Future work will focus on optimizing the computational footprint for real-time edge deployment and testing the architecture on more chaotic physical systems like fluid dynamics.

Summary updated

BibTeX
@inproceedings{tang2023swinlstm,
  author = {Song Tang and Chuang Li and Pu Zhang and Rongnian Tang},
  title = {{SwinLSTM: Improving Spatiotemporal Prediction Accuracy using Swin Transformer and LSTM}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01239},
}
Figure