Towards neural networks that provably know when they don't know

Alexander Meinke, Matthias Hein

Abstract

It has recently been shown that ReLU networks produce arbitrarily over-confident predictions far away from the training data. Thus, ReLU networks do not know when they don't know. However, this is a highly important property in safety critical applications. In the context of out-of-distribution detection (OOD) there have been a number of proposals to mitigate this problem but none of them are able to make any mathematical guarantees. In this paper we propose a new approach to OOD which overcomes both problems. Our approach can be used with ReLU networks and provides provably low confidence predictions far away from the training data as well as the first certificates for low confidence predictions in a neighborhood of an out-distribution point. In the experiments we show that state-of-the-art methods fail in this worst-case setting whereas our model can guarantee its performance while retaining state-of-the-art OOD performance.

Introduction

Deep learning models, specifically those utilizing ReLU activation functions, possess a significant architectural flaw: they are prone to producing arbitrarily high-confidence predictions for inputs that lie far outside their training distribution. In safety-critical applications such as autonomous driving or medical diagnostics, it is imperative that a model "knows when it doesn't know." Current Out-of-Distribution (OOD) detection methods like ODIN or Deep Ensembles are purely empirical; while they perform well on standard benchmarks, they lack mathematical guarantees and can be easily deceived by adversarial OOD examples—noise images specifically perturbed to trigger high-confidence classifications.

The Certified Certain Uncertainty (CCU) Framework

To address this, we introduce Certified Certain Uncertainty (CCU), a probabilistic framework that provides formal guarantees for uncertainty. Instead of relying on standard softmax outputs, CCU models the conditional probability of a class given an input using Bayes' rule, incorporating explicit density estimation.

Explicit Density with GMMs

We utilize Gaussian Mixture Models (GMMs) to estimate the density of in-distribution data. While modern generative models like VAEs or GANs are more expressive, they often share the overconfidence issues of ReLU networks. GMMs are chosen for their well-understood mathematical properties, which allow us to strictly bound their behavior far from cluster centroids. By damping the classifier's confidence using the ratio of in-distribution to out-distribution density, the model is forced to output uniform confidence (maximum uncertainty) far from the training data.

Comparison of confidence landscapes: A standard ReLU network (left) shows overconfidence far from the data, while the CCU framework (right) restricts high confidence to the training distribution.

Experimental Evaluation

We evaluated CCU against several state-of-the-art OOD detection methods, including Outlier Exposure (OE), Mahalanobis distance, and Deep Ensembles (DE), across datasets such as MNIST, SVHN, and CIFAR-10/100.

Adversarial Robustness and OOD Performance

A key contribution of this work is the demonstration that empirical methods fail in worst-case scenarios. When subjected to an adversarial attack designed to maximize confidence within a small radius of uniform noise, standard methods often drop to a 0% success rate. In contrast, CCU achieves a 100% success rate and 100% AUC on these tasks by mathematical design, without sacrificing accuracy on standard in-distribution test sets.

Quantitative results across multiple datasets. CCU consistently achieves a 0.0 Success Rate (SR) for attackers and 100% AUC on adversarial OOD tasks, outperforming empirical methods.

Conclusion

CCU represents a significant step toward provably safe machine learning. By integrating GMM-based density estimation with neural classifiers, we provide the first certificates for low-confidence predictions. While currently limited by the representational power of GMMs on high-dimensional data, this framework establishes a foundation for future research into generative models that can provide strict behavioral guarantees for safety-critical AI.

Summary updated

Abstract

It has recently been shown that ReLU networks produce arbitrarily over-confident predictions far away from the training data. Thus, ReLU networks do not know when they don't know. However, this is a highly important property in safety critical applications. In the context of out-of-distribution detection (OOD) there have been a number of proposals to mitigate this problem but none of them are able to make any mathematical guarantees. In this paper we propose a new approach to OOD which overcomes both problems. Our approach can be used with ReLU networks and provides provably low confidence predictions far away from the training data as well as the first certificates for low confidence predictions in a neighborhood of an out-distribution point. In the experiments we show that state-of-the-art methods fail in this worst-case setting whereas our model can guarantee its performance while retaining state-of-the-art OOD performance.
BibTeX
@inproceedings{meinke2020towards,
  author = {Alexander Meinke and Matthias Hein},
  title = {{Towards neural networks that provably know when they don't know}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=34_QG2K53Yj_},
}
Figure