Abstract
Given a visual history, multiple future outcomes for a video scene are equally probable; in other words, the distribution of future outcomes has multiple modes. Multimodality is notoriously hard to handle by standard regressors or classifiers: the former regress to the mean and the latter discretize a continuous high-dimensional output space. In this work, we present stochastic neural network architectures that handle such multimodality through stochasticity: future trajectories of objects, body joints, or frames are represented as deep, non-linear transformations of random (as opposed to deterministic) variables. Such random variables are sampled from simple Gaussian distributions whose means and variances are parametrized by the output of convolutional encoders over the visual history. We introduce novel convolutional architectures for predicting future body joint trajectories that outperform fully connected alternatives. We introduce stochastic spatial transformers through optical flow warping for predicting future frames, which outperform their deterministic equivalents. Training stochastic networks involves an intractable marginalization over stochastic variables. We compare various training schemes that handle such marginalization through straightforward sampling from the prior, conditional variational autoencoders (CVAE), and a proposed K-best-sample loss that penalizes the best prediction under a fixed "prediction budget." We show experimental results on object trajectory prediction, human body joint trajectory prediction, and video prediction under varying future uncertainty, validating quantitatively and qualitatively our architectural choices and training schemes.
Introduction: The "Blurry Future" Problem
Predicting the future from a visual history is fundamentally ambiguous. For instance, a person walking toward an intersection might turn left, turn right, or continue straight. Standard machine learning models, such as regressors, typically minimize the distance between their prediction and a single ground truth. When faced with such multimodality, these models often "hedge their bets" by predicting the average of all possible paths. In video prediction, this leads to the "blurry ghost" effect where objects fade into a gray smudge. This work introduces Conditional Stochastic Networks (CSNs) to represent the full distribution of possible futures, allowing for the sampling of specific, sharp, and diverse outcomes.
Methods: Conditional Stochastic Architectures
The proposed framework places a stochastic (random) layer between a deterministic encoder and decoder.
Architectures
The authors develop specialized architectures for different prediction tasks:
- Conv-indexed Decoders: For human pose prediction, a fully convolutional network is used instead of standard fully-connected layers to preserve spatial context. The model "indexes" into the resulting feature map using known pixel coordinates of body joints to predict specific trajectories.
- Stochastic Spatial Transformers: For video frame prediction, the model predicts a dense optical flow field rather than generating pixels from scratch. It then uses this flow to "warp" the previous frame into the future, preserving the sharpness and texture of the original image.
Training Schemes
Training networks with random variables requires handling an intractable marginalization. Three approaches are compared:
- MCML (Sampling from the prior): Straightforward sampling from the prior distribution.
- VA (Variational Approximation): A Conditional Variational Autoencoder (CVAE) approach using a "recognition network" during training to help the model learn the latent space.
- MCbest (K-best-sample loss): The primary innovation where the model makes $K$ different guesses. The loss function only penalizes the single best guess among them. This encourages the model to cover different possible modes (e.g., "turn left" vs. "turn right") without being punished for plausible guesses that did not occur in the specific training instance.
Experimental Setup
The models were evaluated on three distinct tasks:
- Stanford Drone Dataset: Predicting trajectories of pedestrians, bikers, and vehicles from an overhead view.
- Human3.6M (H3.6M): Predicting 2D trajectories of human body joints.
- Freiburg-Berkeley Motion Segmentation (MoSeg): Predicting future video frames in "in-the-wild" scenarios.
Key Results and Insights
The $K$-best-sample loss proved highly effective for high-uncertainty scenarios, preventing model collapse into a single "safe" mode.
- Object Trajectories: In cases of extreme uncertainty (e.g., predicting from a single frame), the MCbest method significantly outperformed others in Top-4 error, successfully capturing multiple directions of travel as shown in Figure 3.
- Body Joints: The conv-indexed decoder architecture outperformed previous fully-connected baselines by a wide margin. The VA training scheme proved most stable for this task.
- Video Prediction: Stochastic warping produced much sharper frames than direct pixel generation. On the H3.6M dataset, the model could generate diverse, plausible future poses from a single starting frame.
- Uncertainty vs. History: As the visual history length increases, the future becomes more deterministic, and the relative benefit of stochasticity decreases.
Conclusion
Conditional Stochastic Networks provide a robust framework for handling multimodality in motion prediction. By leveraging spatial context through convolutional architectures and encouraging diversity through the $K$-best-sample loss, these models can generate multiple plausible futures that avoid the pitfalls of deterministic regression. Future work includes scaling these models for longer horizons and better handling of unpredictable camera motion.
Summary updated