Pair-wise Layer Attention with Spatial Masking for Video Prediction

Ping Li, Chenhan Zhang, Zheng Yang, Xianghua Xu, Mingli SongSSRN 2023 · OpenReview

TL;DR

This paper introduces PLA-SM, a novel framework for video prediction that enhances future frame generation by coupling low-level visual details with high-level semantic understanding. It achieves this through a "Pair-wise Layer Attention" mechanism and a "Spatial Masking" pre-training strategy, leading to more robust and detailed predictions.

What Problem Does This Paper Address?

Predicting future frames in a video sequence is a challenging task with applications ranging from autonomous driving to weather forecasting. A key limitation of existing methods is their tendency to lose fine-grained texture details as they process deeper semantic features, resulting in blurry or inaccurate predictions. This happens because current approaches often struggle to effectively integrate low-level visual cues (like edges and textures) with high-level motion trends, especially when dealing with complex scenes and long-term predictions. The paper aims to bridge this gap by improving how spatial features are learned and how different layers of the network interact.

Methods

The proposed PLA-SM framework consists of three main components: an Encoder, a Translator, and a Decoder.

  • Encoder: This part extracts spatial features from input frames. For pre-training, it utilizes a novel Spatial Masking (SM) strategy. Instead of masking entire frames, SM selectively masks portions of the feature maps generated by the encoder. This is done using an attention mechanism that identifies and masks less informative feature pixels, forcing the model to focus on and reconstruct the remaining ones. This pre-training step enhances the robustness of the learned encoding features.
  • Translator: This is the core of the framework, responsible for capturing spatiotemporal dynamics. It employs a U-Net-like architecture with stacked ConvNeXt blocks and a novel Pair-wise Layer Attention (PLA) module. The PLA module is designed to couple low-level visual cues (from earlier layers) with high-level semantic features (from later layers) through an attention mechanism. This allows the model to enrich the semantic features with fine-grained texture details, crucial for realistic predictions.
  • Decoder: This component reconstructs the predicted future frames from the enhanced spatiotemporal features generated by the Translator.

The Spatial Masking module is only used during pre-training, while the Pair-wise Layer Attention module is used during both pre-training and inference.

Datasets

The paper evaluates the PLA-SM framework on five widely used benchmarks:

  • Moving MNIST: A synthetic dataset of evolving handwritten digits.
  • TaxiBJ: Real-world traffic flow data.
  • Human3.6M: A large-scale dataset for 3D human pose estimation, used here for video prediction.
  • KITTI&Caltech Pedestrian: Datasets focused on urban driving scenarios.
  • KTH: A dataset containing human actions like walking, jogging, and boxing.

Key Insights and Ideas

  • Masking Feature Maps, Not Just Pixels: The Spatial Masking strategy, by masking feature maps rather than raw pixels during pre-training, proves more effective in enhancing encoder robustness and feature diversity for CNN-based architectures. This is a key departure from standard masked autoencoder approaches.
  • Bridging Low-Level and High-Level Features: The Pair-wise Layer Attention mechanism is crucial for integrating fine-grained texture details with high-level motion trends. By allowing earlier layers to "attend" to later layers and vice-versa, it ensures that predictions retain visual fidelity.
  • U-Net for Spatiotemporal Dynamics: The adoption of a U-Net-like structure in the Translator, with symmetric encoder-decoder paths for feature processing, helps in capturing and propagating spatiotemporal information effectively.
  • Pre-training for Generalization: The spatial masking pre-training strategy significantly improves the generalization ability of the model, as demonstrated by its performance on datasets with different sources for training and testing (like Caltech Pedestrian).

Key Results

  • State-of-the-Art Performance: PLA-SM consistently outperforms existing state-of-the-art methods across multiple benchmarks (Moving MNIST, TaxiBJ, Human3.6M, Caltech Pedestrian, KTH) in terms of metrics like MSE, MAE, and SSIM.
  • Significant Improvements: On Moving MNIST, TaxiBJ, and Human3.6M, PLA-SM improved performance over the second-best method (SimVP) by 5.4%, 12.3%, and 1.1% in MSE, MAE, and SSIM, respectively.
  • Efficiency: The method achieves competitive or superior performance with significantly lower computational costs (FLOPs) and higher inference speed compared to many previous approaches, particularly those relying on heavy 3D convolutions. For instance, it reduced FLOPs to 9.6G and achieved 310 fps inference.
  • Qualitative Improvements: Visualizations show that PLA-SM generates sharper digits, more accurate traffic flow predictions, and fewer obscured areas in human action sequences, indicating better preservation of texture and motion details.

Limitations and Future Work

  • Fixed Mask Ratio: The paper notes that the fixed mask ratio during pre-training might limit the further increase in robustness of encoding features. The authors suggest exploring adaptive masking strategies to increase feature diversity.
  • Symmetric U-Shape: The current U-shape structure in the Translator is symmetric. The authors propose investigating an asymmetric structure for future work, which might better capture the different contributions of low-level and high-level representations.

Summary updated

Abstract

Video prediction yields future frames by employing the historical frames and has exhibited its great potential in many applications, e.g., meteorological prediction, and autonomous driving. Previous works often decode the ultimate high-level semantic features to future frames without texture details, which deteriorates the prediction quality. Motivated by this, we develop a Pair-wise Layer Attention (PLA) module to enhance the layer-wise semantic dependency of the feature maps derived from the U-shape structure in Translator, by coupling low-level visual cues and high-level features. Hence, the texture details of predicted frames are enriched. Moreover, most existing methods capture the spatiotemporal dynamics by Translator, but fail to sufficiently utilize the spatial features of Encoder. This inspires us to design a Spatial Masking (SM) module to mask partial encoding features during pretraining, which adds the visibility of remaining feature pixels by Decoder. To this end, we present a Pair-wise Layer Attention with Spatial Masking (PLA-SM) framework for video prediction to capture the spatiotemporal dynamics, which reflect the motion trend. Extensive experiments and rigorous ablation studies on five benchmarks demonstrate the advantages of the proposed approach. The code is available at GitHub.
BibTeX
@misc{li2023pair,
  author = {Ping Li and Chenhan Zhang and Zheng Yang and Xianghua Xu and Mingli Song},
  title = {{Pair-wise Layer Attention with Spatial Masking for Video Prediction}},
  year = {2023},
  eprint = {2311.11289},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.11289},
}
Figure