Abstract
We present an approach for pixel-level future prediction given an input image of a scene. We observe that a scene is comprised of distinct entities that undergo motion and present an approach that operationalizes this insight. We implicitly predict future states of independent entities while reasoning about their interactions, and compose future video frames using these predicted states. We overcome the inherent multi-modality of the task using a global trajectory-level latent random variable, and show that this allows us to sample diverse and plausible futures. We empirically validate our approach against alternate representations and ways of incorporating multi-modality. We examine two datasets, one comprising of stacked objects that may fall, and the other containing videos of humans performing activities in a gym, and show that our approach allows realistic stochastic video prediction across these diverse settings.
Introduction
Predicting the future from a single static image is a fundamental challenge in computer vision. Humans excel at this by identifying discrete entities—such as objects or limbs—and understanding their physical properties and potential interactions. For instance, seeing a leaning stack of blocks immediately suggests a likely collapse. Traditional video prediction models often struggle with this task because they attempt to predict the entire pixel space at once, often leading to blurry results or temporal inconsistency as they average out all possible futures.
This research introduces a "compositional" framework for stochastic video prediction. By factorizing a scene into independent entities and reasoning about their trajectories, the model can generate sharp, physically plausible, and diverse future sequences.
Methodology: A Compositional Approach
The proposed framework operates on the principle that scenes are composed of interacting entities. The architecture is divided into three core modules:
1. Entity Predictor
Instead of predicting raw pixels directly, the model predicts the future state (location and appearance features) of each entity. It employs Interaction Networks, a form of Graph Neural Network, to model the dependencies between entities. This allows the model to learn, for example, how a supporting block influences the stability of those above it.
2. Frame Decoder
The decoder reconstructs the full video frame by composing the predicted entity states with the original background. It uses a spatial transformer network to warp features to their predicted locations. Crucially, the model predicts soft masks for each entity, enabling it to handle complex occlusions by determining the relative depth of objects.
3. Global Latent Representation
To address the multi-modal nature of the future (e.g., a stack could fall in multiple directions), the model utilizes a global latent variable ($u$) for the entire video sequence. This variable is processed by an LSTM to generate per-timestep variables ($z_t$), ensuring long-term temporal consistency. This prevents "jittery" or physically inconsistent motions where an object might change direction abruptly between frames.
Evaluation and Results
The model was rigorously tested on two distinct datasets: ShapeStacks, featuring synthetic 3D blocks, and Penn Action, featuring real-world gym activities where entities are defined as human joints.
The results demonstrate several key advantages:
- Generalization: By learning interaction rules rather than scene-specific layouts, a model trained on small stacks of blocks can successfully predict the behavior of much larger stacks.
- Emergent Segmentation: The model learns to produce accurate instance segmentation masks without explicit mask supervision, relying only on bounding box locations and pixel reconstruction.
- Temporal Coherence: The use of a global latent variable significantly reduces the "flickering" common in other stochastic models, maintaining object identity and motion direction throughout the sequence.
Quantitatively, the compositional approach outperformed baselines like SVG-LP in perceptual quality (LPIPS) and location accuracy. Qualitatively, it produced sharper objects that retained their physical properties over time, avoiding the "melting" effect seen in non-factorized models.
Conclusion
Compositional video prediction offers a robust path toward more realistic and physically grounded future forecasting. By treating scenes as collections of interacting entities and enforcing global temporal consistency, this framework overcomes major hurdles in high-dimensional, multi-modal prediction. Future work aims to reduce the reliance on initial entity annotations through unsupervised discovery and to extend the model to scenes with dynamic backgrounds and more complex physical phenomena.
Summary updated