TL;DR
Autonomous driving systems need to anticipate the future, but standard deterministic models fail when the future is inherently uncertain (such as turning a corner). This paper introduces a multimodal semantic forecasting approach using conditional Generative Adversarial Networks (GANs) combined with moment reconstruction losses to predict multiple plausible future scenes. The resulting model successfully captures diverse potential outcomes without sacrificing short-term accuracy.
What Problem Does This Paper Address?
Semantic forecasting—predicting the pixel-level classification of future video frames—is crucial for autonomous vehicles to plan safe trajectories. However, most existing methods treat this as a deterministic regression problem, assuming a single, entirely certain future.
When faced with genuine ambiguity (e.g., a car turning a corner or a large vehicle unoccluding a street), a deterministic model can only predict the statistical average or most likely outcome. This behavior often erases smaller, critical objects like pedestrians or traffic signs, because the model prefers a safe "background" guess to minimize its regression error. The authors address this limitation by enabling multimodal forecasting: allowing the model the freedom to generate a distribution of multiple distinct, plausible futures.
Methods
The authors propose MM-DeformF2F, a multimodal framework built on top of a single-level feature-to-feature (F2F) forecasting model (DeformF2F). Instead of predicting a single future, they transform the architecture into a conditional generative model using three main components:
- Generator with Noise Injection: They inject Gaussian noise tensors ($z_{1:K}$) into the F2F module to generate $K$ different future feature maps conditioned on a blend of features extracted from past frames.
- Moment Reconstruction (MR) Loss: To combat "mode collapse" (where a GAN generator outputs identical samples regardless of the noise input), they replace traditional $L_1/L_2$ reconstruction losses with Moment Reconstruction losses (MR1 and MR2). These losses force the model to match the statistical moments (mean and variance) of the predicted distribution to the ground truth, directly encouraging output diversity.
- PatchGAN Discriminator: A modified PatchGAN discriminator evaluates whether the generated abstract feature maps look real or fake. Dropout is introduced in its first layer to prevent the discriminator from overpowering the generator during training.
Datasets
The model was trained and evaluated on the Cityscapes dataset, a standard benchmark for urban driving scenes:
- The dataset contains video sequences representing 2,975 training scenes, 500 validation scenes, and 1,525 testing scenes across 19 semantic classes.
- Each scene consists of 30 frames (1.8 seconds total duration), with ground-truth semantic segmentation available for the 20th frame.
- To manage the computational complexity introduced by the GAN architecture, all images were halved in width and height to reduce the feature footprint and speed up training.
Key Insights and Ideas
- The Accuracy vs. Diversity Trade-off: Using the MR1 loss (which optimizes only for the mean) yields high accuracy on par with or slightly better than deterministic baselines. Conversely, the MR2 loss (which incorporates variance) significantly boosts the diversity of the generated futures but causes a noticeable drop in standard mean Intersection over Union (mIoU) metrics.
- Averaging Fails the Concept: While averaging the $K$ generated predictions together improves the overall mIoU score, the authors note this contradicts the core philosophy of multimodal forecasting, which aims to explore distinct edge-case scenarios rather than a single blurred average.
- Visualizing Uncertainty: By computing the variance across discrete predictions, the model naturally highlights regions of high environmental ambiguity (e.g., boundaries of moving vehicles or newly unoccluded spaces), providing a built-in confidence map for downstream driving decisions.
Key Results
- Short-Term Forecasting ($t+3$ frames): The proposed MM-DeformF2F with MR1 loss achieved a mean Intersection over Union (mIoU) of 59.22% (and 59.46% when averaged), outperforming the deterministic DeformF2F baseline (58.98%).
- Moving Objects: Accuracy on moving objects (mIoU-MO) improved from 56.00% (baseline) to 56.40% (MR1) and 56.66% (MR1 averaged).
- Diversity Metrics: With the MR2 loss, diversity skyrocketed—Mean Squared Error (MSE) between predictions rose from 1.21 (MR1) to 4.38 (MR2), and LPIPS increased from 0.0482 to 0.1519, though at the cost of lower absolute mIoU (53.85%).
- Computational Efficiency: Generating 8 predictions ($K=8$) only added about 15 seconds of overhead to a 500-image evaluation batch on a GTX980 GPU compared to the baseline, proving that multimodal generation does not require a linear scale-up in processing time.
Limitations and Future Work
- The MR2 Performance Penalty: The MR2 loss introduces excellent diversity but degrades semantic segmentation accuracy (mIoU drops by ~5 percentage points). Finding a sweet spot that preserves both remains an open challenge.
- Hesitation on Large Objects: Inspection of the generated forecasts shows that the model is still hesitant to alter large, close-up objects, focusing its diversity mostly on smaller, distant, or highly dynamic objects like bicycles and pedestrians.
- Future Directions: The authors suggest exploring proxy MR losses for more stable early-stage training, incorporating discriminators that leverage global contextual information, and extending the framework to instance-level segmentation forecasting.
Summary updated