Stochastic Generative Models for Multi-Modal Video Prediction

This direction is merged into another direction: stochastic future prediction. Its last version stays readable and citable here.

Version 1 of 141 papers47 conceptsfrozen 2026-10-04

When we predict the future, we are rarely certain. If you watch a person walking toward a fork in a path, they might turn left or right. A deterministic model trained to minimize Mean Squared Error (MSE) will attempt to predict the "average" of these two possibilities. The result is a ghostly, translucent figure that splits in two or a single blurry blob that disappears into the middle of the path.

This "blurry average" problem has been the central hurdle in video prediction. Research has shifted from simple regression toward stochastic generative models that can sample from a distribution of many possible futures.

The Failure of Deterministic Regression

Early attempts at video prediction treated the task as a standard regression problem. However, as Michaël Mathieu and Yann LeCun observed in Deep Multi Scale Video Prediction Beyond Mean Square Error, pixel-wise loss functions like MSE assume a unimodal distribution. When the future is inherently multi-modal—meaning there are multiple valid, distinct outcomes—minimizing the squared error forces the model to output the mean of all potential trajectories.

To combat this, Mathieu et al. introduced Adversarial Training (GANs) and Image Gradient Difference Loss (GDL). By forcing the model to "fool" a discriminator, the model is incentivized to pick one sharp, realistic future rather than an average of many.

Capturing Uncertainty with Latent Variables

If the blurriness is caused by uncertainty, the logical solution is to model that uncertainty explicitly. Chelsea Finn and Sergey Levine pioneered this with Stochastic Variational Video Prediction (SV2P).

SV2P uses a Variational Autoencoder (VAE) framework to learn a latent distribution of possible motions. At inference time, the model samples a value from this latent space, which acts as a "seed" for the specific future it will generate. This allows the same starting frames to produce multiple, distinct, and sharp realizations. Lluís Castrejón and Aaron Courville furthered this in Improved Conditional VRNNs for Video Prediction, arguing that blurriness is often a symptom of underfitting that can be resolved by increasing the capacity of the latent hierarchy.

Structural Priors and Intermediate Representations

Sometimes, modeling raw pixels is too difficult because the search space is too large. Some researchers use intermediate "bridges" to simplify the uncertainty. In The Pose Knows, Jacob Walker and Abhinav Gupta first predict the future of a human skeleton (pose) before generating pixels. Because pose is a low-dimensional representation, the model can more easily navigate the multi-modal possibilities of human movement before committing to a high-resolution visual output.

Similarly, Gunhee Kim and Yale Song proposed disentangling appearance from motion in Video Prediction with Appearance and Motion Conditions. By separating "what" an object is from "where" it is going, the model reduces the complexity of the prediction task, preventing the appearance of the object from blurring as its position becomes uncertain.

From Autoregression to Diffusion

The field has evolved through several generative paradigms to better handle this stochasticity:

  • Autoregressive Pixel Modeling: Aäron van den Oord and Oriol Vinyals developed Video Pixel Networks (VPNs), which model the joint distribution of pixels as a dependency chain. By predicting one pixel at a time, the model avoids independence assumptions that lead to blur.
  • Normalizing Flows: VideoFlow by Laurent Dinh and Durk Kingma uses invertible transformations to directly optimize data likelihood, allowing for faster and more stable stochastic sampling than pixel-level autoregression.
  • Diffusion Models: Most recently, Alexia Jolicoeur-Martineau and Christopher Pal introduced MCVD (Masked Conditional Video Diffusion). Diffusion models bypass the need for explicit latent hierarchies by learning to denoise a signal, naturally capturing complex, multi-modal distributions without the training instability of GANs.

As summarized by Jingwei Wu in A Survey on Video Prediction, the field is moving away from simple frame synthesis toward "world models." These models don't just predict the next frame; they simulate the underlying stochastic laws of the environment, ensuring that every predicted future is a sharp, physically plausible possibility.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_stochastic_generative_models_for_multi_m_007fc214,
  title = {Stochastic Generative Models for Multi-Modal Video Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stochastic-generative-models-for-multi-modal-video-prediction/rdr_007fc21424004b88b1a1a62c9c1d2a42},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_stochastic_generative_models_for_multi_m_007fc214_v1,
  title = {Stochastic Generative Models for Multi-Modal Video Prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stochastic-generative-models-for-multi-modal-video-prediction/rdr_007fc21424004b88b1a1a62c9c1d2a42/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}