Abstract
Spatiotemporal predictive learning (ST-PL) aims at predicting the subsequent frames via limited observed sequences, and it has broad applications in the real world. However, learning representative spatiotemporal features for prediction is challenging. Moreover, chaotic uncertainty among consecutive frames exacerbates the difficulty in long-term prediction. This paper concentrates on improving prediction quality by enhancing the correspondence between the previous context and the current state. We carefully design Detail Context Block (DCB) to extract fine-grained details and improve the isolated correlation between upper context state and current input state. We integrate DCB with standard ConvLSTM and introduce Motion Details RNN (MoDeRNN) to capture fine-grained spatiotemporal features and improve the expression of latent states of RNNs to achieve significant quality. Experiments on Moving MNIST and Typhoon datasets demonstrate the effectiveness of the proposed method. MoDeRNN outperforms existing state-of-the-art techniques qualitatively and quantitatively with lower computation loads.
Introduction
Predicting future video frames from past observations—known as Spatiotemporal Predictive Learning (ST-PL)—is notoriously difficult due to chaotic motion and dynamic visual changes. Mainstream approaches usually rely on stacked Recurrent Neural Networks (RNNs) or Convolutional LSTMs (ConvLSTMs).
However, as these models get deeper to handle long-term predictions, the correlation between the current input frame and the "upper context" (the hidden state from previous steps) degrades. In standard ConvLSTMs, these states only interact via simple channel-wise addition and standard CNN layers. This isolated, weak interaction fails to capture the fine-grained, localized motion details necessary for accurate forecasting, causing future frames to quickly degrade into a blurry mess, especially when objects overlap or trajectories change.
Methods
The authors retain the standard Encoder-RNN-Decoder architecture but completely revamp the internal RNN cell, creating the Motion Details RNN (MoDeRNN).
The core innovation is the Detail Context Block (DCB). Instead of just adding the input state ($X_t$) and the context state ($H_{t-1}$) together, the DCB forces them to interact iteratively through mutual attention:
- Context-to-Input Attention: It uses multi-kernel CNNs (with varying receptive fields like 3x3, 5x5, and 7x7) to generate an attention weight map from the context state. This map is used to reweight the input state via a Hadamard (element-wise) product, highlighting the most important parts of the input.
- Input-to-Context Attention: It performs the reverse operation, generating an attention map from the input state to update the context state, ensuring the hidden memory is enriched with current, fine-grained motion trends.
By stacking these DCBs inside the RNN cell, MoDeRNN ensures that fine-grained local features—like the sharp edges of a moving object or the eye of a storm—are explicitly preserved and passed along to future timestamps.
Experiments and Results
The authors evaluated their model on two standard benchmarks: Moving MNIST, a synthetic dataset featuring digits moving with constant velocity, and Typhoon, a real-world meteorological radar dataset.
Key Results
- Moving MNIST: MoDeRNN achieved a PSNR of 22.472 and SSIM of 0.936, improving upon the previous state-of-the-art (SA-ConvLSTM) by 9.62% and 2.52% respectively. It also reduced Mean Squared Error (MSE) by 12.03%. Qualitatively, it successfully maintained clarity even when digits overlapped.
- Typhoon: MoDeRNN achieved the best performance across all metrics (PSNR: 29.446, SSIM: 0.910), successfully preserving the detailed textures of meteorological movements over time better than any baseline.
- Parameter Efficiency: MoDeRNN achieved these SOTA results with only 4.59 Million parameters. For comparison, competing models are significantly heavier: SA-ConvLSTM (10.47M), PredRNN (13.79M), and E3D-LSTM (38.69M).
Conclusion
MoDeRNN demonstrates that dense state interaction is crucial for long-term spatiotemporal prediction. By forcing the input and context states to mutually weight each other through the multi-scale DCB, the model preserves sharp edges and prevents the blurring typical of standard RNN predictions. This architecture proves that smarter, more deliberate interaction blocks can vastly outperform much larger, heavier networks in both quality and efficiency.
Summary updated