When we predict the future, we are rarely certain. If you watch a person walking toward a fork in a path, they might turn left or right. A deterministic model trained to minimize Mean Squared Error (MSE) will attempt to predict the "average" of these two possibilities. The result is a ghostly, translucent figure that splits in two or a single blurry blob that disappears into the middle of the path.
This "blurry average" problem has been the central hurdle in video prediction. Research has shifted from simple regression toward stochastic generative models that can sample from a distribution of many possible futures.
The Failure of Deterministic Regression
Early attempts at video prediction treated the task as a standard regression problem. However, as Michaël Mathieu and Yann LeCun observed in Deep Multi Scale Video Prediction Beyond Mean Square Error, pixel-wise loss functions like MSE assume a unimodal distribution. When the future is inherently multi-modal—meaning there are multiple valid, distinct outcomes—minimizing the squared error forces the model to output the mean of all potential trajectories.
To combat this, Mathieu et al. introduced Adversarial Training (GANs) and Image Gradient Difference Loss (GDL). By forcing the model to "fool" a discriminator, the model is incentivized to pick one sharp, realistic future rather than an average of many.
Capturing Uncertainty with Latent Variables
If the blurriness is caused by uncertainty, the logical solution is to model that uncertainty explicitly. Chelsea Finn and Sergey Levine pioneered this with Stochastic Variational Video Prediction (SV2P).
SV2P uses a Variational Autoencoder (VAE) framework to learn a latent distribution of possible motions. At inference time, the model samples a value from this latent space, which acts as a "seed" for the specific future it will generate. This allows the same starting frames to produce multiple, distinct, and sharp realizations. Lluís Castrejón and Aaron Courville furthered this in Improved Conditional VRNNs for Video Prediction, arguing that blurriness is often a symptom of underfitting that can be resolved by increasing the capacity of the latent hierarchy.
Structural Priors and Intermediate Representations
Sometimes, modeling raw pixels is too difficult because the search space is too large. Some researchers use intermediate "bridges" to simplify the uncertainty. In The Pose Knows, Jacob Walker and Abhinav Gupta first predict the future of a human skeleton (pose) before generating pixels. Because pose is a low-dimensional representation, the model can more easily navigate the multi-modal possibilities of human movement before committing to a high-resolution visual output.
Similarly, Gunhee Kim and Yale Song proposed disentangling appearance from motion in Video Prediction with Appearance and Motion Conditions. By separating "what" an object is from "where" it is going, the model reduces the complexity of the prediction task, preventing the appearance of the object from blurring as its position becomes uncertain.
From Autoregression to Diffusion
The field has evolved through several generative paradigms to better handle this stochasticity:
- Autoregressive Pixel Modeling: Aäron van den Oord and Oriol Vinyals developed Video Pixel Networks (VPNs), which model the joint distribution of pixels as a dependency chain. By predicting one pixel at a time, the model avoids independence assumptions that lead to blur.
- Normalizing Flows: VideoFlow by Laurent Dinh and Durk Kingma uses invertible transformations to directly optimize data likelihood, allowing for faster and more stable stochastic sampling than pixel-level autoregression.
- Diffusion Models: Most recently, Alexia Jolicoeur-Martineau and Christopher Pal introduced MCVD (Masked Conditional Video Diffusion). Diffusion models bypass the need for explicit latent hierarchies by learning to denoise a signal, naturally capturing complex, multi-modal distributions without the training instability of GANs.
As summarized by Jingwei Wu in A Survey on Video Prediction, the field is moving away from simple frame synthesis toward "world models." These models don't just predict the next frame; they simulate the underlying stochastic laws of the environment, ensuring that every predicted future is a sharp, physically plausible possibility.