What is Flagged in Uncertainty Quantification? Latent Density Models for Uncertainty Categorization

Hao Sun, Boris van Breugel, Jonathan Crabbé, Nabeel Seedat, Mihaela van der Schaar

TL;DR

The DAUC (Density-based Approach for Uncertainty Categorization) framework moves beyond simply flagging "uncertain" predictions by explaining why they are suspicious. By introducing the "confusion density matrix," the authors categorize model uncertainty into three distinct types—Out-of-Distribution (OOD), Boundary (Bnd), and In-Distribution Misclassification (IDM)—providing a new benchmark for evaluating and improving uncertainty quantification (UQ) methods.

The "Why" Behind Model Uncertainty

Modern machine learning models, especially deep neural networks, are often treated as "black boxes." While existing uncertainty quantification (UQ) methods like Deep Ensembles or MC-Dropout can successfully flag a prediction as "untrusted," they rarely illuminate the root cause of that uncertainty.

Is the model uncertain because it has never seen this type of data before (Out-of-Distribution, or OOD)? Is the example sitting right on the "fence" between two classes (Boundary)? Or is the model simply prone to making mistakes in that specific region of the data distribution (In-Distribution Misclassification, or IDM)? Without knowing the "why," it is difficult for engineers to decide on the best corrective action—whether that means collecting more data, refining the model architecture, or handing the decision over to a human expert. This paper fills that gap by providing a granular, post-hoc categorization of uncertainty.

The DAUC framework pipeline and latent space categorization of untrusted predictions into OOD, Boundary, and IDM.

The DAUC Framework and Confusion Density Matrix

To address this, the authors propose DAUC, a model-agnostic framework that operates on the model's latent representation space (the feature layer just before the final classification).

The core innovation of DAUC is the Confusion Density Matrix. Unlike a standard confusion matrix that counts global errors, this matrix uses Kernel Density Estimation (KDE) to measure the local "density" of different types of examples around a new test point. It calculates three primary scores:

  • OOD Score: Measures how far a test point is from the entire training manifold. A low density indicates the point is Out-of-Distribution.
  • Boundary (Bnd) Score: Measures the density of correctly classified validation examples from different classes nearby. High density here suggests the point is near a decision boundary.
  • IDM Score: Measures the density of misclassified validation examples nearby. High density suggests the model is fundamentally "confused" in this region.

By comparing these scores against learned thresholds, DAUC can label any flagged example as OOD, Bnd, IDM, or a combination (such as Boundary & IDM).

Experimental Setup and Datasets

The researchers evaluated DAUC using a mix of synthetic and real-world benchmarks designed to stress-test uncertainty categorization:

  • Two-Smiles: A modified version of the "Two-Moons" dataset used for 2D visualization of uncertainty types.
  • Dirty-MNIST: A combination of vanilla MNIST (digits), Ambiguous-MNIST (synthesized boundary cases), and Fashion-MNIST (used as OOD data).
  • Dirty-CIFAR-10: A newly introduced benchmark created by using linear combinations of latent representations to create "boundary" classes, with MNIST digits serving as OOD examples.
  • UCI Benchmarks: Covtype, Digits, and Spam datasets were used to demonstrate how DAUC can actively improve model performance.

Key Insights and Results

The application of DAUC revealed several critical insights into how different UQ methods behave:

  • UQ Methods are Highly Specialized: One of the most striking findings is that different UQ methods "see" different things. For example, Deep Ensembles (DE) are exceptionally good at flagging OOD samples as the most uncertain. In contrast, MC-Dropout is surprisingly poor at OOD detection, sometimes flagging OOD samples as "highly certain." Bayesian Neural Networks (BNNs) tended to flag a more even mix of OOD, Boundary, and IDM cases.
  • Latent Space is Key: The authors demonstrate that performing density estimation in the latent space is superior to using the raw input space. Latent spaces are lower-dimensional and more closely coupled to the model’s actual decision-making process, making the density scores significantly more meaningful.
  • High Categorization Accuracy: On the Dirty-MNIST benchmark, DAUC achieved near-perfect F1-scores (1.0 for OOD, 0.96 for Bnd, and 0.92 for IDM) in identifying exactly why an example was flagged.
  • Actionable Uncertainty Leads to Performance Gains: Categorization allows for targeted model improvement. By using DAUC to identify the "hardest" examples (the Boundary & IDM class) and retraining a linear model on a filtered subset of training data most relevant to those cases, the authors achieved a linear increase in prediction accuracy.

Limitations and Future Directions

While powerful, the framework does have limitations. It assumes the model can be decomposed into a feature extractor and a linear classifier, which may not apply to all non-parametric models. Additionally, while KDE is efficient for low-dimensional latent spaces, scaling these calculations to extremely large datasets or very high-dimensional embeddings could introduce computational bottlenecks.

Looking forward, the authors suggest extending DAUC beyond classification tasks. Adapting the framework for regression and Reinforcement Learning (RL)—where understanding the source of uncertainty is critical for exploration—is a vital next step. Furthermore, applying DAUC to identify "suspicious" outputs in Large Language Models (LLMs) holds significant promise for advancing AI alignment and safety.

Summary updated

Abstract

Uncertainty quantification (UQ) is essential for creating trustworthy machine learning models. Recent years have seen a steep rise in UQ methods that can flag suspicious examples, however, it is often unclear what exactly these methods identify. In this work, we propose a framework for categorizing uncertain examples flagged by UQ methods. We introduce the confusion density matrix---a kernel-based approximation of the misclassification density---and use this to categorize suspicious examples identified by a given uncertainty method into three classes: out-of-distribution (OOD) examples, boundary (Bnd) examples, and examples in regions of high in-distribution misclassification (IDM). Through extensive experiments, we show that our framework provides a new and distinct perspective for assessing differences between uncertainty quantification methods, thereby forming a valuable assessment benchmark.
BibTeX
@inproceedings{sun2023what,
  author = {Hao Sun and Boris van Breugel and Jonathan Crabbé and Nabeel Seedat and Mihaela van der Schaar},
  title = {{What is Flagged in Uncertainty Quantification? Latent Density Models for Uncertainty Categorization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=zyhxRc9bew},
}
Figure