Bayesian Prediction of Future Street Scenes through Importance Sampling based Optimization

Apratim Bhattacharyya, Mario Fritz, Bernt SchielearXiv 2018 · DOI · arXiv

TL;DR

This paper introduces a novel Bayesian approach to predicting future street scenes, specifically focusing on semantic segmentation. By treating deep learning models as Bayesian, they can capture both model and observation uncertainty, leading to more robust and calibrated predictions, especially for long-term forecasting. Their key innovation is an importance sampling scheme that leverages a recognition network to efficiently train these complex Bayesian models.

What Problem Does This Paper Address?

Autonomous agents, like self-driving cars, need to anticipate future events to operate safely. Predicting future "street scenes" – essentially, what objects will be where in the next few moments – is crucial. While current deep learning models can predict future semantic segmentation (identifying objects like cars, pedestrians, roads), they often struggle with uncertainty. As the prediction horizon extends, the future becomes increasingly uncertain and can have multiple plausible outcomes (multi-modal). Existing deterministic models can produce highly confident but incorrect predictions, which is dangerous. This paper aims to address this by providing a framework that explicitly models and quantifies this uncertainty, offering a more trustworthy prediction of future scenes.

Methods

The core idea is to cast state-of-the-art deep learning models for semantic segmentation into a Bayesian framework. This allows them to model two types of uncertainty:

  • Observation Uncertainty (Aleatoric): This accounts for inherent noise and variability in the observed data itself. The model predicts the mean and variance of a Gaussian distribution for each pixel's class prediction.
  • Model Uncertainty (Epistemic): This accounts for uncertainty in the model's parameters. Instead of learning a single set of weights, they learn a distribution over possible weights.

To handle the intractability of full Bayesian inference, they employ variational inference. A key innovation is a novel importance sampling scheme that uses an auxiliary recognition network. This recognition network learns to propose good "samples" of model parameters, making the training process more efficient and stable, especially for multi-modal distributions. This is inspired by techniques used in Variational Autoencoders (VAEs) but applied to sampling model weights rather than latent codes. They also incorporate weight dropout as a form of approximate Bayesian inference.

Datasets

The primary dataset used for training and evaluation is the Cityscapes dataset. This is a widely used benchmark for semantic segmentation in urban street scenes. They also mention using the annotated 20th frame of validation sequences for evaluation.

Key Insights and Ideas

  • Bayesian modeling is crucial for uncertainty: Simply predicting the most likely future scene isn't enough. Understanding how uncertain the prediction is, and whether there are multiple plausible futures, is vital for safe decision-making by autonomous agents.
  • Importance sampling with a recognition network is key to training: Directly training complex Bayesian models with many parameters is computationally challenging. Their proposed importance sampling scheme, guided by a learned recognition network, significantly improves training efficiency and stability.
  • Weight dropout is more effective than standard dropout for Bayesian inference: While standard dropout can approximate Bayesian inference, their experiments suggest that applying dropout to weights (rather than activations) and using it within their importance sampling framework leads to better results.
  • Capturing multi-modality is essential for long-term prediction: As the prediction horizon increases, street scenes become more diverse and less predictable. Their Bayesian approach, particularly with importance sampling, is better at capturing these multiple possible futures compared to deterministic models.

Key Results

  • State-of-the-art performance: Their proposed Bayesian model (Bayes-WD-IS) achieves state-of-the-art results on the Cityscapes dataset for predicting future semantic segmentation, outperforming previous methods in terms of mean Intersection-over-Union (mIoU). For example, they show a 5.6 mIoU improvement at +0.06sec over a strong baseline.
  • Well-calibrated uncertainty: The model's predicted probabilities for class occurrences closely match the actual frequencies, indicating reliable uncertainty estimates. This is measured by the Conditional Log-Likelihood (CLL) metric, where their model shows significant improvements over baselines, especially for longer prediction horizons (e.g., 1.04 CLL at +1sec vs. 1.68 for a strong baseline).
  • Improved multi-modal prediction: Qualitative results show their model can better capture diverse outcomes for objects like pedestrians and cars, especially in challenging scenarios.

Limitations and Future Work

  • Computational cost: While their importance sampling scheme improves efficiency, Bayesian deep learning models are generally more computationally intensive than their deterministic counterparts.
  • Future work: The paper suggests that their methodology can be extended to other sequence prediction tasks and that marrying high-performance deep learning architectures with principled Bayesian inference is a promising avenue for future research. They also highlight the potential for their well-calibrated uncertainty estimates to enable better control and decision-making for autonomous systems.

Summary updated

Abstract

For autonomous agents to successfully operate in the real world, anticipation of future events and states of their environment is a key competence. This problem can be formalized as a sequence prediction problem, where a number of observations are used to predict the sequence into the future. However, real-world scenarios demand a model of uncertainty of such predictions, as future states become increasingly uncertain and multi-modal -- in particular on long time horizons. This makes modelling and learning challenging. We cast state of the art semantic segmentation and future prediction models based on deep learning into a Bayesian formulation that in turn allows for a full Bayesian treatment of the prediction problem. We present a new sampling scheme for this model that draws from the success of variational autoencoders by incorporating a recognition network. In the experiments we show that our model outperforms prior work in accuracy of the predicted segmentation and provides calibrated probabilities that also better capture the multi-modal aspects of possible future states of street scenes.
BibTeX
@misc{bhattacharyya2018bayesian,
  author = {Apratim Bhattacharyya and Mario Fritz and Bernt Schiele},
  title = {{Bayesian Prediction of Future Street Scenes through Importance Sampling based Optimization}},
  year = {2018},
  eprint = {1806.06939},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.06939},
}
Figure