Uncertainty-based out-of-distribution detection requires suitable function space priors

Francesco D'Angelo, Christian Henning

Abstract

The question whether inputs are valid for the problem a neural network is trying to solve has sparked interest in out-of-distribution (OOD) detection. It is widely assumed that Bayesian neural networks (BNNs) are well suited for this task, as the endowed epistemic uncertainty should lead to disagreement in predictions on outliers. In this paper, we question this assumption and show that proper Bayesian inference with function space priors induced by neural networks does not necessarily lead to good OOD detection. To circumvent the use of approximate inference, we start by studying the infinite-width case, where Bayesian inference can be exact due to the correspondence with Gaussian processes. Strikingly, the kernels derived from common architectural choices lead to function space priors which induce predictive uncertainties that do not reflect the underlying input data distribution and are therefore unsuited for OOD detection. Importantly, we find the OOD behavior in this limiting case to be consistent with the corresponding finite-width case. To overcome this limitation, useful function space properties can also be encoded in the prior in weight space, however, this can currently only be applied to a specified subset of the domain and thus does not inherently extend to OOD data. Finally, we argue that a trade-off between generalization and OOD capabilities might render the application of BNNs for OOD detection undesirable in practice. Overall, our study discloses fundamental problems when naively using BNNs for OOD detection and opens interesting avenues for future research.

Introduction

In safety-critical AI applications, a model must possess the ability to signal when it encounters data that differs significantly from its training set. This capability, known as Out-of-Distribution (OOD) detection, is often attributed to Bayesian Neural Networks (BNNs). The prevailing theory suggests that BNNs naturally model epistemic uncertainty—the uncertainty stemming from a lack of data—which should theoretically increase when the model is presented with outliers.

However, this paper challenges the notion that BNNs are inherently suited for OOD detection. The authors demonstrate that the quality of OOD detection is not a byproduct of Bayesian inference itself, but rather a consequence of the function-space prior induced by the chosen architecture and weight priors. If the prior does not naturally favor uncertainty in OOD regions, the resulting posterior will fail to provide reliable safety signals.

Methodology: The Infinite-Width Limit

To isolate the influence of the prior from the potential errors introduced by approximate inference methods (such as Variational Inference), the researchers focus on the infinite-width limit of BNNs. In this regime, BNNs converge to Neural Network Gaussian Processes (NNGPs), allowing for exact Bayesian inference using kernel functions derived from specific architectures.

Figure 1: Gaussian Process samples and mean prediction relative to training data points.

By comparing these infinite-width models with finite-width networks trained using Hamiltonian Monte Carlo (HMC), the authors confirm that the OOD behavior observed in the limiting case is consistent with practical, finite-width implementations. This suggests that the "prior problem" is a fundamental architectural characteristic.

The Role of Architecture as a Prior

The choice of activation function is, in essence, a choice of prior. The study reveals that standard ReLU-based BNNs do not behave like Kernel Density Estimators. In low-dimensional spaces, ReLU kernels exhibit an "angular" dependence, meaning they can remain highly confident even as an input moves far away from the training data in certain directions.

Conversely, architectures like RBF-networks or Cosine networks induce kernels that behave more like the Squared Exponential (RBF) kernel, where uncertainty naturally increases as the distance from the training data manifold increases.

Figure 4: Uncertainty landscape for an infinite-width RBF network on a 2D synthetic dataset.

Key Insights and Trade-offs

One of the most significant findings is the Generalization vs. OOD Trade-off. A prior specifically designed to improve generalization (e.g., a periodic prior for periodic data) can actually degrade OOD detection. By baking in domain-specific knowledge, the model may be forced to be "certain" about regions of the input space where it has never observed data, leading to dangerous overconfidence on outliers.

Furthermore, the "Curse of Dimensionality" poses a significant hurdle for high-dimensional data like images. Even "good" kernels (like RBF) struggle on benchmarks like SplitMNIST because Euclidean distance is a poor metric for similarity in complex image manifolds. The authors found that points of lowest uncertainty were sometimes OOD images, while some in-distribution test points exhibited higher uncertainty than outliers.

Conclusion

This research discloses fundamental limitations in the naive application of BNNs for OOD detection. The effectiveness of these models is tethered to the suitability of their function-space priors. Future work must move beyond simple Euclidean metrics and focus on developing priors that inherently understand the underlying manifold of the data to provide reliable uncertainty estimates in high-dimensional settings.

Summary updated

Abstract

The question whether inputs are valid for the problem a neural network is trying to solve has sparked interest in out-of-distribution (OOD) detection. It is widely assumed that Bayesian neural networks (BNNs) are well suited for this task, as the endowed epistemic uncertainty should lead to disagreement in predictions on outliers. In this paper, we question this assumption and show that proper Bayesian inference with function space priors induced by neural networks does not necessarily lead to good OOD detection. To circumvent the use of approximate inference, we start by studying the infinite-width case, where Bayesian inference can be exact due to the correspondence with Gaussian processes. Strikingly, the kernels derived from common architectural choices lead to function space priors which induce predictive uncertainties that do not reflect the underlying input data distribution and are therefore unsuited for OOD detection. Importantly, we find the OOD behavior in this limiting case to be consistent with the corresponding finite-width case. To overcome this limitation, useful function space properties can also be encoded in the prior in weight space, however, this can currently only be applied to a specified subset of the domain and thus does not inherently extend to OOD data. Finally, we argue that a trade-off between generalization and OOD capabilities might render the application of BNNs for OOD detection undesirable in practice. Overall, our study discloses fundamental problems when naively using BNNs for OOD detection and opens interesting avenues for future research.
BibTeX
@inproceedings{dangelo2022uncertainty,
  author = {Francesco D'Angelo and Christian Henning},
  title = {{Uncertainty-based out-of-distribution detection requires suitable function space priors}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=u7UxOTefG2},
}
Figure