Multimodal semantic forecasting based on conditional generation of future features

Kristijan Fugošić, Josip Šarić, Siniša ŠegvićarXiv 2020 · arXiv

TL;DR

Autonomous driving systems need to anticipate the future, but standard deterministic models fail when the future is inherently uncertain (such as turning a corner). This paper introduces a multimodal semantic forecasting approach using conditional Generative Adversarial Networks (GANs) combined with moment reconstruction losses to predict multiple plausible future scenes. The resulting model successfully captures diverse potential outcomes without sacrificing short-term accuracy.

What Problem Does This Paper Address?

Semantic forecasting—predicting the pixel-level classification of future video frames—is crucial for autonomous vehicles to plan safe trajectories. However, most existing methods treat this as a deterministic regression problem, assuming a single, entirely certain future.

When faced with genuine ambiguity (e.g., a car turning a corner or a large vehicle unoccluding a street), a deterministic model can only predict the statistical average or most likely outcome. This behavior often erases smaller, critical objects like pedestrians or traffic signs, because the model prefers a safe "background" guess to minimize its regression error. The authors address this limitation by enabling multimodal forecasting: allowing the model the freedom to generate a distribution of multiple distinct, plausible futures.

Methods

The authors propose MM-DeformF2F, a multimodal framework built on top of a single-level feature-to-feature (F2F) forecasting model (DeformF2F). Instead of predicting a single future, they transform the architecture into a conditional generative model using three main components:

  1. Generator with Noise Injection: They inject Gaussian noise tensors ($z_{1:K}$) into the F2F module to generate $K$ different future feature maps conditioned on a blend of features extracted from past frames.
  2. Moment Reconstruction (MR) Loss: To combat "mode collapse" (where a GAN generator outputs identical samples regardless of the noise input), they replace traditional $L_1/L_2$ reconstruction losses with Moment Reconstruction losses (MR1 and MR2). These losses force the model to match the statistical moments (mean and variance) of the predicted distribution to the ground truth, directly encouraging output diversity.
  3. PatchGAN Discriminator: A modified PatchGAN discriminator evaluates whether the generated abstract feature maps look real or fake. Dropout is introduced in its first layer to prevent the discriminator from overpowering the generator during training.
Fig. 1: Structural diagram of our model. Base structure is similar to [19]. It is composed of feature extractor (yellow), upsampling branch (blue), spatial pyra- mid pooling (SPP) and F2F. Additionally, we introduce random noise z, dis- criminator D and new loss functions. Our model generates multiple predictions.

Datasets

The model was trained and evaluated on the Cityscapes dataset, a standard benchmark for urban driving scenes:

  • The dataset contains video sequences representing 2,975 training scenes, 500 validation scenes, and 1,525 testing scenes across 19 semantic classes.
  • Each scene consists of 30 frames (1.8 seconds total duration), with ground-truth semantic segmentation available for the 20th frame.
  • To manage the computational complexity introduced by the GAN architecture, all images were halved in width and height to reduce the feature footprint and speed up training.

Key Insights and Ideas

  • The Accuracy vs. Diversity Trade-off: Using the MR1 loss (which optimizes only for the mean) yields high accuracy on par with or slightly better than deterministic baselines. Conversely, the MR2 loss (which incorporates variance) significantly boosts the diversity of the generated futures but causes a noticeable drop in standard mean Intersection over Union (mIoU) metrics.
  • Averaging Fails the Concept: While averaging the $K$ generated predictions together improves the overall mIoU score, the authors note this contradicts the core philosophy of multimodal forecasting, which aims to explore distinct edge-case scenarios rather than a single blurred average.
  • Visualizing Uncertainty: By computing the variance across discrete predictions, the model naturally highlights regions of high environmental ambiguity (e.g., boundaries of moving vehicles or newly unoccluded spaces), providing a built-in confidence map for downstream driving decisions.

Key Results

  • Short-Term Forecasting ($t+3$ frames): The proposed MM-DeformF2F with MR1 loss achieved a mean Intersection over Union (mIoU) of 59.22% (and 59.46% when averaged), outperforming the deterministic DeformF2F baseline (58.98%).
  • Moving Objects: Accuracy on moving objects (mIoU-MO) improved from 56.00% (baseline) to 56.40% (MR1) and 56.66% (MR1 averaged).
  • Diversity Metrics: With the MR2 loss, diversity skyrocketed—Mean Squared Error (MSE) between predictions rose from 1.21 (MR1) to 4.38 (MR2), and LPIPS increased from 0.0482 to 0.1519, though at the cost of lower absolute mIoU (53.85%).
  • Computational Efficiency: Generating 8 predictions ($K=8$) only added about 15 seconds of overhead to a 500-image evaluation batch on a GTX980 GPU compared to the baseline, proving that multimodal generation does not require a linear scale-up in processing time.

Limitations and Future Work

  • The MR2 Performance Penalty: The MR2 loss introduces excellent diversity but degrades semantic segmentation accuracy (mIoU drops by ~5 percentage points). Finding a sweet spot that preserves both remains an open challenge.
  • Hesitation on Large Objects: Inspection of the generated forecasts shows that the model is still hesitant to alter large, close-up objects, focusing its diversity mostly on smaller, distant, or highly dynamic objects like bicycles and pedestrians.
  • Future Directions: The authors suggest exploring proxy MR losses for more stable early-stage training, incorporating discriminators that leverage global contextual information, and extending the framework to instance-level segmentation forecasting.

Summary updated

Abstract

This paper considers semantic forecasting in road-driving scenes. Most existing approaches address this problem as deterministic regression of future features or future predictions given observed frames. However, such approaches ignore the fact that future can not always be guessed with certainty. For example, when a car is about to turn around a corner, the road which is currently occluded by buildings may turn out to be either free to drive, or occupied by people, other vehicles or roadworks. When a deterministic model confronts such situation, its best guess is to forecast the most likely outcome. However, this is not acceptable since it defeats the purpose of forecasting to improve security. It also throws away valuable training data, since a deterministic model is unable to learn any deviation from the norm. We address this problem by providing more freedom to the model through allowing it to forecast different futures. We propose to formulate multimodal forecasting as sampling of a multimodal generative model conditioned on the observed frames. Experiments on the Cityscapes dataset reveal that our multimodal model outperforms its deterministic counterpart in short-term forecasting while performing slightly worse in the mid-term case.
BibTeX
@misc{fugosic2020multimodal,
  author = {Kristijan Fugošić and Josip Šarić and Siniša Šegvić},
  title = {{Multimodal semantic forecasting based on conditional generation of future features}},
  year = {2020},
  eprint = {2010.09067},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.09067},
}
Figure