Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts

Bertrand Charpentier, Daniel Zügner, Stephan Günnemann

Overview

Deep neural networks are notoriously overconfident. A standard classifier trained exclusively on images of dogs will likely predict "dog" with 99% confidence when shown a picture of a toaster, simply because it lacks a mechanism to say, "I don't know."

The Posterior Network (PostNet) is a novel architecture designed to provide reliable uncertainty estimation in a single forward pass—without ever needing to see out-of-distribution (OOD) samples during training. By combining Normalizing Flows with a Bayesian framework, PostNet learns to assign high "epistemic" uncertainty to any input that falls into low-density regions of its latent space.

The Challenge of Uncertainty

Reliable AI systems must distinguish between two types of uncertainty:

  1. Aleatoric (Data) Uncertainty: Inherent noise in the data, such as a blurry image that could legitimately be a '3' or an '8'.
  2. Epistemic (Knowledge) Uncertainty: A lack of knowledge, such as encountering a category of data never seen during training.

Existing solutions have significant drawbacks. Bayesian Neural Networks and Ensembles are computationally expensive, requiring multiple passes or models to estimate uncertainty. "Prior Networks" are faster but require researchers to provide explicit OOD data (like random noise) during training to teach the model what "unknown" looks like. This is an unrealistic requirement for real-world applications where the "unknown" is inherently unpredictable.

The PostNet Architecture

PostNet addresses these gaps using density-based pseudo-counts. The architecture consists of three main components:

  1. Encoder: A standard neural network maps an input $x$ to a low-dimensional latent vector $z$.
  2. Normalizing Flows: For each class, the model maintains a Normalizing Flow—a powerful generative tool that learns the probability density $P(z|c)$ of that class within the latent space.
  3. Dirichlet Posterior: The densities from the flows are treated as "pseudo-counts" (evidence) for a Dirichlet distribution.
Overview of the PostNet architecture showing how inputs are mapped to a latent space where densities are evaluated to parameterize a Dirichlet distribution.

If an input maps to a region in the latent space where the density for all classes is near zero, the Dirichlet distribution reverts to a flat prior, signaling high epistemic uncertainty. If it maps to a region where multiple class densities overlap, it signals high aleatoric uncertainty. The entire system is trained end-to-end using a novel Bayesian loss function called Uncertain Cross Entropy (UCE).

Key Insights

  • Density as Evidence: The core innovation is using the normalized density of a Normalizing Flow as the parameter for a Dirichlet distribution. This naturally links the concept of "how much data have I seen here?" to "how confident should I be?"
  • Avoiding the Curse of Dimensionality: Instead of performing density estimation on high-dimensional raw pixels—which is notoriously difficult—PostNet operates in a low-dimensional latent space.
  • No OOD Training Required: Because Normalizing Flows are mathematically constrained to integrate to one, they must assign low density to regions far from the training data. This allows PostNet to detect OOD samples "for free."
PostNet correctly identifies high uncertainty (dark regions) in areas far from the training data distribution.

Datasets and Results

The authors evaluated PostNet across a variety of domains, including tabular data (Segment, Sensorless Drive), image data (MNIST, CIFAR-10), and synthetic datasets. To test OOD detection, they used datasets like FashionMNIST and SVHN as "unseen" data.

PostNet achieved state-of-the-art results in OOD detection and uncertainty calibration:

  • OOD Detection: On the Sensorless Drive dataset, PostNet improved OOD detection (measured by AUC-PR) by over 30% absolute compared to previous Dirichlet-based models.
  • Robustness to Shift: On CIFAR-10, PostNet showed the most graceful degradation in confidence as images were increasingly corrupted (e.g., blurred or snowed out), whereas baselines remained stubbornly overconfident.
  • Efficiency: Unlike Ensembles or Dropout-based methods, PostNet requires only a single forward pass, making it highly suitable for real-time applications like autonomous driving or medical monitoring.

Limitations and Future Work

While powerful, PostNet has a few limitations. The model's performance is sensitive to the dimensionality of the latent space; if the dimension is too high, density estimation becomes unreliable. Additionally, training stability requires careful architectural choices, such as adding Batch Normalization after the encoder to ensure latent positions match the density regions.

Future directions include exploring more complex Normalizing Flow architectures and applying the PostNet framework to other tasks like regression or active learning.

Summary updated

Abstract

Accurate estimation of aleatoric and epistemic uncertainty is crucial to build safe and reliable systems. Traditional approaches, such as dropout and ensemble methods, estimate uncertainty by sampling probability predictions from different submodels, which leads to slow uncertainty estimation at inference time. Recent works address this drawback by directly predicting parameters of prior distributions over the probability predictions with a neural network. While this approach has demonstrated accurate uncertainty estimation, it requires defining arbitrary target parameters for in-distribution data and makes the unrealistic assumption that out-of-distribution (OOD) data is known at training time. In this work we propose the Posterior Network (PostNet), which uses Normalizing Flows to predict an individual closed-form posterior distribution over predicted probabilites for any input sample. The posterior distributions learned by PostNet accurately reflect uncertainty for in- and out-of-distribution data -- without requiring access to OOD data at training time. PostNet achieves state-of-the art results in OOD detection and in uncertainty calibration under dataset shifts.
BibTeX
@inproceedings{charpentier2020posterior,
  author = {Bertrand Charpentier and Daniel Zügner and Stephan Günnemann},
  title = {{Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=DgG47SvTOX},
}
Figure