Real-Time Uncertainty Estimation in Computer Vision via Uncertainty-Aware Distribution Distillation

Yichen Shen, Zhilu Zhang, Mert R. Sabuncu, Lin SunWACV 2021 · DOI

TL;DR

This paper introduces a novel "Uncertainty-Aware Distribution Distillation" method that allows deep learning models to estimate uncertainty in real-time for computer vision tasks. By distilling the full predictive distribution from a slower, uncertainty-aware "teacher" model (like MC Dropout) into a faster "student" model, they achieve significant speedups without sacrificing, and often improving, the quality of uncertainty estimates.

What Problem Does This Paper Address?

In critical applications like autonomous driving and medical diagnosis, understanding how confident a deep learning model is in its predictions is as important as the prediction itself. Traditional methods for estimating this uncertainty, such as Monte Carlo (MC) Dropout, are computationally expensive because they require multiple forward passes through the network at inference time. This latency makes them unsuitable for real-time applications. While prior work on "Dropout Distillation" aimed to speed this up, it only learned from the mean predictions of the teacher, neglecting crucial uncertainty information. This paper tackles the challenge of achieving fast, sample-free uncertainty estimation that captures the full predictive distribution.

Methods

The core of the proposed method is a teacher-student distillation framework.

  • Teacher Model: A pre-trained model capable of estimating uncertainty, typically using MC Dropout. This teacher model can also be extended to incorporate aleatoric uncertainty by outputting an estimate of observation noise.
  • Student Model: A deterministic neural network trained to approximate the entire predictive distribution of the teacher model. Instead of just learning the mean prediction, the student learns to output the parameters of a distribution (e.g., Laplace for regression, logit-normal for classification) that captures both epistemic and aleatoric uncertainty.
  • Distillation Process: The student is trained using Maximum Likelihood Estimation (MLE) on samples generated by the teacher. To improve robustness, the student is also trained using ground truth labels.
  • Additional Augmentation: To prevent the student from underestimating epistemic uncertainty due to overfitting, the authors propose using extra data augmentation during student training that was not used for the teacher. This helps the student see teacher predictions that are more aligned with test-time distributions.

Datasets

The proposed method was evaluated on several standard computer vision datasets:

  • Semantic Segmentation:
    • CamVid
    • Pascal VOC2012
  • Depth Estimation:
    • NYU Depth V2 (NYU)
    • KITTI Odometry dataset (KITTI)
  • Distribution Shift Evaluation:
    • Cityscapes (evaluated on overlapped classes with CamVid)

Key Insights and Ideas

  • Distilling the Full Distribution is Key: Simply distilling the mean prediction (as in prior work) discards vital uncertainty information. By learning to parameterize the entire predictive distribution, the student can capture richer uncertainty signals.
  • Real-Time Uncertainty is Achievable: The proposed method allows for sample-free uncertainty estimation, reducing inference time from seconds (for MC Dropout) to milliseconds, making it practical for real-time systems.
  • Student Can Surpass Teacher: In several metrics, the student model not only matches but exceeds the teacher's performance in terms of predictive accuracy and uncertainty quality. This is attributed to the student being supervised by both ground truth and the teacher's distilled distribution.
  • Additional Augmentation for Robustness: Using extra data augmentation during student training, even if not used for the teacher, significantly improves the student's ability to estimate uncertainty, particularly for outlier detection.
  • Outlier Detection Benefits: The method shows promise in identifying out-of-distribution samples by assigning them higher uncertainty, which is crucial for safety-critical applications.

Key Results

  • Speedup: The student model achieves significant speedups, with runtime reduced from seconds (MC Dropout) to milliseconds. For depth estimation on NYU, a speedup ratio of 46x was reported.
  • Performance:
    • Semantic Segmentation (CamVid): The student model achieved comparable or better accuracy and IOU than the teacher, with significantly lower ECE and AUSE (better uncertainty calibration). For example, the student achieved an AUSE of 1.60x10^-2 compared to the teacher's 1.47x10^-2.
  • Depth Estimation (NYU/KITTI): The student consistently outperformed the teacher across RMSE, REL, and log10 metrics, and also showed improved ECE and AUSE. For NYU, the student achieved an ECE of 8.09x10^-2 compared to the teacher's 9.38x10^-2.
  • Qualitative Results: Visual examples show the student model producing uncertainty maps that closely match the teacher's, often highlighting subtle anomalies or out-of-distribution regions with high uncertainty.
  • Distribution Shift: When evaluated on the Cityscapes dataset (unseen during training), the student model performed significantly better than the teacher, suggesting enhanced robustness.

Limitations and Future Work

  • Teacher Dependence: The student's performance is inherently tied to the quality of the teacher model.
  • Hyperparameter Tuning: The distillation process involves hyperparameters (e.g., the weight $\lambda$ for combining losses, number of samples $m, k$) that need tuning.
  • Aleatoric vs. Epistemic Overlap: The paper notes that incorporating aleatoric uncertainty might lead to minimal improvements in some cases due to potential overlap between the two uncertainty types. However, they emphasize the generality of their approach.
  • Future Work: The authors suggest that their method is applicable to a wider range of tasks and datasets, and that further exploration of the benefits of aleatoric uncertainty in different scenarios is warranted. The improved robustness under distribution shift also opens avenues for further research in domain adaptation and generalization.

Summary updated

BibTeX
@inproceedings{shen2021real,
  author = {Yichen Shen and Zhilu Zhang and Mert R. Sabuncu and Lin Sun},
  title = {{Real-Time Uncertainty Estimation in Computer Vision via Uncertainty-Aware Distribution Distillation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00075},
}
Figure