Abstract
Generating videos predicting the future of a given sequence has been an area of active research in recent years. However, an essential problem remains unsolved: most of the methods require large computational cost and memory usage for training. In this paper, we propose a novel method for generating future prediction videos with less memory usage than the conventional methods. This is a critical stepping stone in the path towards generating videos with high image quality, similar to that of generated images in the latest works in the field of image generation. We achieve high-efficiency by training our method in two stages: (1) image reconstruction to encode video frames into latent variables, and (2) latent variable prediction to generate the future sequence. Our method achieves an efficient compression of video into low-dimensional latent variables by decomposing each frame according to its hierarchical structure. That is, we consider that video can be separated into background and foreground objects, and that each object holds time-varying and time-independent information independently. Our experiments show that the proposed method can efficiently generate future prediction videos, even for complex datasets that cannot be handled by previous methods.
1. Introduction
Predicting future video frames from a set of initial frames is a significant computational challenge. State-of-the-art methods typically rely on 3D CNNs or complex recurrent architectures that process entire video sequences simultaneously. This approach often leads to a "memory wall," restricting research to low resolutions or simplified datasets. While disentangled representations—separating "content" from "pose"—have been explored, they often lack the granularity required for complex scenes with multiple moving objects and detailed backgrounds.
This paper introduces a two-stage training framework that shifts temporal prediction from high-dimensional pixel space to a structured, low-dimensional latent space, significantly reducing memory and computational overhead.
2. Methodology
The proposed approach decomposes the video generation task into two distinct phases: hierarchical image reconstruction and latent variable sequence prediction.
2.1 Stage 1: Hierarchical Image Reconstruction
The model first learns to compress individual frames into a set of disentangled latent variables. The architecture is split into two sub-networks:
- Background VAE: Learns to reconstruct static backgrounds by swapping latent variables between different frames of the same video during training.
- Object VAE: Utilizes an LSTM-based detector to identify foreground objects. Each object is decomposed into three specific latent variables: Position ($z_{where}$), Pose ($z_{pose}$), and Content ($z_{content}$).
2.2 Stage 2: Latent Variable Sequence Prediction
Once the VAE is trained, the video dataset is converted into sequences of low-dimensional latent vectors. A secondary LSTM-based model is then trained to predict future latent variables based on past observations. By operating entirely in the latent space, this stage avoids the memory costs associated with raw pixel processing.
3. Disentanglement and Latent Space Analysis
A key innovation of this framework is the fine-grained disentanglement of object properties. By explicitly separating position from pose, the model avoids re-learning an object's appearance simply to move it. To ensure $z_{content}$ captures time-invariant features, the authors employ a "swap and mix" strategy during training, mixing content variables from different frames of the same video.
4. Experimental Results
The method was evaluated on two primary datasets: a complex version of Moving MNIST (featuring CIFAR-10 backgrounds and color-changing digits) and the CMU Motion Capture dataset.
- Visual Quality: On complex Moving MNIST, the method maintained sharp digit shapes and correct colors over 16 predicted frames, whereas baselines often produced blurry results or failed to capture foreground colors.
- Robustness: In the CMU dataset, the model generated consistent human figures over long horizons, while competing methods like MoCoGAN and DRNET showed increasing blurriness.
- Efficiency: Memory usage for the prediction stage was reduced from $O(BT HWC)$ to approximately $O(BHW C + BT k)$, where $k$ is the small dimension of the latent variables.
5. Conclusion
This hierarchical framework provides a memory-efficient path toward high-quality video generation. While the model currently requires a predefined maximum number of objects and can occasionally produce slightly blurry backgrounds, it significantly outperforms existing baselines in complex scenarios. Future work will explore applying this architecture to unconditional video generation and video interpolation.
Summary updated