Robust uncertainty estimates with out-of-distribution pseudo-inputs training

Pierre Segonne, Yevgen Zainchkovskyy, Søren Hauberg

Abstract

Neural networks frequently exhibit overconfidence when encountering out-of-distribution (OOD) data, posing significant risks for real-world deployment. This paper introduces a "dissipative loss" framework that utilizes a Pseudo-Input Generator (PIG) to create artificial data points at the boundaries of the training distribution. By explicitly training the model to output high uncertainty at these pseudo-inputs, the method achieves robust, state-of-the-art uncertainty quantification for regression and generative models. Crucially, a decoupled training strategy ensures that OOD robustness is achieved without degrading standard predictive performance.

Introduction

For machine learning models to be safely deployed in high-stakes environments, they must provide well-calibrated uncertainty estimates. Ideally, when a model encounters an input significantly different from its training data—an out-of-distribution (OOD) input—it should flag the prediction with high uncertainty.

However, standard neural networks tend to extrapolate arbitrarily outside their training support. A network trained to predict variance might confidently output low uncertainty for completely alien data. While existing solutions like Deep Ensembles or Bayesian methods exist, they often come with high computational costs or inadvertently degrade the model's core predictive accuracy by entangling the mean and variance components of the loss function.

Methods: The Dissipative Loss Framework

The authors propose a novel approach built upon the Variational Variance (VV) framework, introducing a training objective called Dissipative Loss powered by a Pseudo-Input Generator (PIG).

Pseudo-Input Generation (PIG)

Since training data for OOD regions is by definition unavailable, the authors generate artificial "pseudo-inputs." The PIG algorithm starts by sampling from the existing training data and uses gradient descent to push these points into low-density regions, effectively placing them just outside the boundaries of the known data distribution.

Algorithm 1: The Pseudo-Input Generator (PIG) procedure.

Dissipative Loss and Split Training

The core idea is to "fence in" the model's predictions. While the model is trained to maximize standard likelihood on real data, a KL-divergence penalty is applied to the pseudo-inputs. This forces the predicted variance at these boundary points to match a high-uncertainty prior (e.g., a broad Gamma distribution).

Visualization of training with pseudo-inputs (circles) versus standard data (diamonds).

To prevent this OOD regularization from interfering with the model's primary task, the authors employ a Split Training procedure:

  1. The network's mean predictions are trained first until accurate.
  2. The mean network is frozen.
  3. The variance (uncertainty) network is then trained using both real data and the generated pseudo-inputs.

This decoupling ensures that the model maintains state-of-the-art accuracy while gaining robust uncertainty estimates.

Key Results

The proposed method, referred to as d-VV, was evaluated across regression benchmarks and generative modeling tasks.

Regression Performance

On 12 standard UCI Machine Learning Repository datasets, including versions with artificial distributional shifts, d-VV outperformed competitive baselines such as Deep Ensembles, MC Dropout, and Bayes by Backprop. It achieved the best Evidence Lower Bound (ELBO) on nearly all tested datasets, demonstrating superior robustness to shifts.

Generative Modeling and OOD Detection

When applied to Variational Autoencoders (creating the d-V3AE), the dissipative loss regularizes the decoder's variance in the latent space. This results in:

  • Higher Quality Generations: Qualitatively smoother and more realistic images compared to standard Gaussian VAEs.
  • Superior OOD Detection: In a test where a model trained on FashionMNIST was exposed to MNIST data, the d-V3AE achieved a near-perfect AUROC of 0.992, significantly higher than the standard VAE's 0.857.

Conclusion

The dissipative loss approach provides a computationally efficient (~26% overhead) path to robust uncertainty. By focusing regularization on the boundaries of the training data rather than the entire input space, the model learns to "know what it doesn't know" effectively.

A current limitation is the reliance on a differentiable density estimate to guide the PIG. While VAEs provide this naturally via the aggregate posterior, regression tasks require an initial density estimation step (e.g., using a Gaussian Mixture Model). Future research may focus on integrating more direct density estimation techniques into the training pipeline to further generalize the approach.

Summary updated

BibTeX
@inproceedings{segonne2021robust,
  author = {Pierre Segonne and Yevgen Zainchkovskyy and Søren Hauberg},
  title = {{Robust uncertainty estimates with out-of-distribution pseudo-inputs training}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=YJL3amnDf2w},
}
Figure