Bayesian Prediction of Future Street Scenes using Synthetic Likelihoods

Apratim Bhattacharyya, Mario Fritz, Bernt Schiele

Introduction

Predicting the future state of complex environments, such as city streets for autonomous driving, is a fundamental challenge in computer vision. A key difficulty is multi-modality: for any given past sequence, multiple plausible futures exist. For instance, a pedestrian at a corner might cross the street or remain on the sidewalk. Standard Bayesian deep learning methods often struggle with this, as their objective functions tend to "average out" these possibilities, resulting in blurry, unrealistic predictions that fail to capture the true diversity of potential outcomes.

The Multi-modality Problem

Current Bayesian methods, particularly those utilizing Dropout for approximate inference, typically rely on maximizing log-likelihood. This objective forces every sampled model configuration to explain every data point in the training set. When faced with multi-modal ground truth, models "hedge their bets," leading to a regression to the mean.

Comparison of standard dropout (top) vs. synthetic likelihoods (bottom) on a multi-modal 1D task. Standard methods average the modes, while the proposed method captures them.

As illustrated above, standard dropout (top) converges to a single, non-representative mean between two modes. In contrast, the proposed synthetic likelihood approach (bottom) allows the model to accurately represent distinct possible outcomes.

Proposed Method: Synthetic Likelihoods

To address the limitations of standard log-likelihood, this paper introduces a framework combining Bayesian inference with generative adversarial principles. The core innovations include:

  1. Weight-based Dropout: Unlike traditional patch-based dropout in CNNs, the authors apply dropout directly to the weights. This reduces the complexity of the model configuration space while effectively capturing model uncertainty.
  2. Synthetic Likelihood (SL): Instead of pixel-wise loss, a discriminator is trained to distinguish between real future scenes and generated ones. The Bayesian KL-divergence objective is reformulated to use the ratio provided by this discriminator. This allows a generated sample to be considered "correct" if it is plausible under the true data distribution, even if it doesn't match a specific ground truth instance perfectly.
  3. Hybrid Objective: A small amount of traditional log-likelihood is combined with the synthetic likelihood to ensure training stability.
  4. Architecture: The model uses a 15-layer ResNet-based encoder-decoder that processes past semantic segmentations and vehicle odometry to recursively predict future semantic maps.

Experimental Results

The model was evaluated across diverse tasks to demonstrate its ability to capture multi-modality and maintain calibration.

Cityscapes and MNIST

On the Cityscapes benchmark, the proposed "Bayes-WD-SL" model achieved state-of-the-art performance, reaching 52.5 mIoU at a 0.54s horizon, significantly outperforming previous methods. In "Oracle" tests (evaluating the top 5% of samples), the accuracy increased further, confirming the model captures the correct future as one of its hypotheses.

MNIST digit completion results and quantitative comparison. The proposed Bayes-SL model captures multiple modes (e.g., '3' and '5') and achieves the highest Top-10% accuracy.

In the MNIST completion task, where models must predict the rest of a digit from a partial quadrant, the Bayes-SL model successfully generated diverse, realistic completions (e.g., turning a partial stroke into either a '3' or a '5'). It achieved 83.1% Top-10% accuracy, surpassing Conditional Variational Autoencoders (CVAEs).

Precipitation Forecasting

Testing on the HKO Radar Echo Dataset showed the model's generalizability. It outperformed specialized meteorological models in forecasting rainfall patterns, achieving a lower Rainfall-MSE (1.033) and a high correlation (0.945).

Conclusion and Future Work

By relaxing the constraints of standard log-likelihood through synthetic likelihoods, this approach enables Bayesian models to handle multi-modal data without losing the benefits of probabilistic calibration. While the model currently faces challenges with fine-grained resolution due to heavy pooling and computational overhead during sampling, it represents a significant step toward realistic, diverse future prediction in autonomous systems. Future work will focus on extending these predictions to longer time horizons.

Summary updated

BibTeX
@inproceedings{bhattacharyya2019bayesian,
  author = {Apratim Bhattacharyya and Mario Fritz and Bernt Schiele},
  title = {{Bayesian Prediction of Future Street Scenes using Synthetic Likelihoods}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=mjMq6AoxBt},
}
Figure