TL;DR
This paper compares different neural network architectures (FCNN, CRNN, CLSTM) and training strategies (stateless vs. stateful) for predicting the next frame in a video. While FCNNs offer the highest prediction quality (PSNR), they are computationally expensive. Recurrent networks (CRNN/CLSTM) with stateful training provide a compelling trade-off, achieving near real-time performance with acceptable quality and significantly lower computational cost.
What Problem Does This Paper Address?
Predicting future frames in a video is a crucial task with applications ranging from video compression to generating realistic animations. While deep learning has shown promise in this area, there's been a lack of systematic comparison between different architectural choices (like feedforward vs. recurrent networks) and how training methodologies impact their performance and efficiency. This paper aims to fill that gap by evaluating the effectiveness of Fully Convolutional Neural Networks (FCNNs) against Recurrent Neural Networks (RNNs), specifically Convolutional RNNs (CRNNs) and Convolutional LSTMs (CLSTMs), under different training regimes.
Methods
The authors investigate three main neural network architectures:
- Residual Fully Convolutional Neural Network (FCNN): Inspired by the EDSR architecture for super-resolution, this network processes multiple input frames to predict a single output frame. It's characterized by its deep structure with many residual blocks.
- Convolutional RNN (CRNN) and Convolutional LSTM (CLSTM): These are recurrent architectures designed to capture temporal dependencies in video sequences. The paper highlights that CRNNs, often plagued by training instability, can be made stable and efficient using a specific training technique.
Two training strategies are explored for the recurrent networks:
- Stateless Training: This involves truncating backpropagation through time (TBPTT) after a fixed number of frames (8 in this case). Each training sequence is treated independently, and recurrent states are reset at the beginning of each sequence. This requires storing gradients for multiple frames, leading to higher memory usage.
- Stateful Training: This method uses TBPTT with a truncation size of 1, meaning backpropagation occurs one step at a time. Crucially, the recurrent states are not reset between frames within a minibatch. Instead, they are carried over, allowing the network to learn from longer temporal contexts without excessive memory overhead. This also leads to more frequent parameter updates.
The primary loss function used for training is the mean square loss, chosen to optimize for Peak Signal-to-Noise Ratio (PSNR).
Datasets
- Training: The authors used two million grayscale patch sequences extracted from the UCF101 dataset [16]. These sequences were 48x48 pixels and 9 frames long, with the task being to predict the 9th frame given the first 8.
- Testing: Eight MPEG sequences were used for evaluation: Coastguard, Container, Football, Foreman, Garden, Hall Monitor, Mobile, and Tennis. While training used cropped patches, the evaluation predicted full frames.
Key Insights and Ideas
- FCNNs excel at quality, but at a cost: The FCNN architecture consistently achieved the highest PSNR, indicating superior prediction accuracy. However, this comes at the significant expense of higher computational complexity during both training and inference.
- Stateful training is a game-changer for RNNs: For CRNNs and CLSTMs, stateful training dramatically improves convergence speed and allows for near real-time inference (17 fps) with acceptable performance. This is a significant advantage over stateless training, which is slower to converge and requires more memory.
- CRNNs can be viable with the right training: By employing stateful truncated backpropagation through time, the CRNN, which is typically harder to train than CLSTM, becomes a stable and efficient option.
- Camera motion is a challenge: Videos with significant camera motion (like "garden" and "mobile") proved more difficult for the recurrent models, suggesting that more parameters might be needed to capture complex motion.
- Memory efficiency matters: Stateful training's lower memory footprint is a key advantage, enabling the training of larger models and potentially allowing for larger batch sizes for more robust training.
Key Results
- PSNR Performance: The FCNN achieved the highest PSNR across all test videos. Statefully trained CRNN and CLSTM models provided competitive results, often close to the FCNN, especially on videos without extreme camera motion.
- Inference Speed:
- FCNN: 1 frame per second (fps).
- CRNN/CLSTM (stateful training): 17 fps (near real-time).
- Training Convergence: Stateful training converged significantly faster than stateless training for both CRNN and CLSTM models.
- GPU Memory Load: Stateful training required substantially less GPU memory (e.g., CRNN: 1783 MB vs. 3353 MB for stateless) compared to stateless training. FCNN training required a substantial 12 GB.
Limitations and Future Work
- Blurry Predictions: The paper notes that using mean square loss can lead to blurry predicted images.
- Future Work: The authors suggest that the proposed architectures and training procedures could be adapted for use with other loss functions (e.g., KL-divergence, adversarial loss, l¹ loss, gradient loss, perceptual loss) to emphasize visual detail and sharpness, depending on the specific application. This opens avenues for improving the perceptual quality of predicted frames.
Summary updated