Inception LSTM for Next-frame Video Prediction (Student Abstract)

Matin Hosseini, Anthony S. Maida, Majid Hosseini, Raju GottumukkalaAAAI 2020 · DOI

Abstract

Standard Convolutional LSTMs (ConvLSTMs) are often constrained by fixed kernel sizes, creating a trade-off between capturing fine-grained spatial details and large-scale temporal motion. This paper proposes the Inception LSTM, an architecture that integrates multi-scale Inception modules into LSTM gates. By processing features across various spatial resolutions simultaneously, the model improves next-frame video prediction. Evaluations on the KITTI and KTH datasets demonstrate that Inception LSTM variants consistently outperform standard ConvLSTMs in terms of MAE, MSE, and SSIM, often achieving better results with fewer layers.

Introduction

Video frame prediction is a fundamental challenge in autonomous navigation, robotics, and video compression. The goal is to generate the next frame in a sequence by learning both spatial features and temporal dynamics. Current state-of-the-art models typically utilize Convolutional LSTMs (ConvLSTMs). However, a standard ConvLSTM uses a fixed kernel size (e.g., 3x3) for all its gates, which limits its receptive field. A small kernel may capture fine textures but struggle with fast-moving objects, while a large kernel captures broad motion at the expense of high-resolution detail.

The Inception LSTM aims to overcome this "receptive field bottleneck" by allowing the model to "see" at multiple scales within each temporal step, effectively capturing motion at different speeds and objects at different scales.

Methodology

The proposed Inception LSTM replaces the single-kernel convolution in each LSTM gate—input, forget, cell, and output—with an Inception-inspired module. Two primary variants were developed:

  • Inception v1 LSTM: Employs three parallel kernels of sizes 1x1, 3x3, and 5x5. The outputs are concatenated before the activation function.
  • Inception v2 LSTM: Replaces the 5x5 kernel with two stacked 3x3 kernels to reduce computational overhead while maintaining the same receptive field.

These cells were integrated into the PredNet predictive coding framework. To optimize for LSTM gate operations, the authors modified the standard Inception module by removing max-pooling and 1x1 bottleneck layers.

Architecture of the Inception LSTM cell integrated with a predictive coding framework.

Experimental Setup

The models were evaluated on two benchmark datasets:

  1. KITTI: Real-world traffic sequences from a moving vehicle, characterized by complex backgrounds and varying object speeds.
  2. KTH: Human action sequences featuring six distinct activities (e.g., walking, boxing).

Performance was measured using Mean Absolute Error (MAE), Mean Squared Error (MSE), and the Structural Similarity Index (SSIM).

Results and Discussion

The Inception-based models demonstrated superior performance across both datasets compared to the baseline ConvLSTM.

Quantitative comparison of ConvLSTM and Inception LSTM variants on KITTI and KTH datasets.

Key Findings

  • Multi-Scale Motion Capture: Different kernel sizes act as varying "shutter speeds." Small kernels handle local changes, while larger or stacked kernels track rapid, global motion.
  • Breadth vs. Depth: An Inception LSTM with 3 layers outperformed a standard 4-layer ConvLSTM. This indicates that increasing the "width" (feature diversity) of a layer can be more effective than simply adding depth.
  • Efficiency: While Inception v1 generally yielded the highest accuracy, Inception v2 provided a computationally efficient alternative with minimal performance loss.
  • Training Speed: The proposed model achieved competitive results in only 50 training epochs, significantly faster than the 150 epochs typically required for standard PredNet.

Conclusion and Future Work

The Inception LSTM effectively addresses the receptive field limitations of standard ConvLSTMs by incorporating multi-scale feature extraction. While these modules increase architectural complexity, the gains in predictive accuracy and training efficiency are substantial. Future work will explore broader hyperparameter tuning for kernel sizes and investigate the impact of longer training cycles and the specific role of pooling within recurrent gates.

Summary updated

BibTeX
@inproceedings{hosseini2020inception,
  author = {Matin Hosseini and Anthony S. Maida and Majid Hosseini and Raju Gottumukkala},
  title = {{Inception LSTM for Next-frame Video Prediction (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7176},
}
Figure