Out-of-Distribution Detection with Class Ratio Estimation

Mingtian Zhang, Andi Zhang, Tim Z. Xiao, Yitong Sun, Steven McDonagharXiv 2022 · DOI · arXiv

Abstract

This paper introduces a simplified approach to Out-of-Distribution (OOD) detection by reframing it as a class ratio estimation problem. By utilizing a binary classifier to estimate density ratios between in-distribution data and a background base distribution, the proposed method avoids the computational complexity of deep generative models while achieving state-of-the-art performance. A "spread density ratio" technique using Gaussian noise further stabilizes training, ensuring robust OOD scores across diverse benchmarks.

Introduction

In machine learning, ensuring that models can identify samples from outside their training distribution is critical for safety and reliability. Traditional approaches often rely on generative models to calculate the likelihood of a sample; however, these models frequently suffer from a "likelihood paradox," where they assign higher probabilities to simple OOD samples (like SVHN digits) than to complex in-distribution data (like CIFAR-10 images).

To address this, recent research has shifted toward using density ratios, comparing the input's density against a "base" distribution. While effective, these methods typically require training complex generative models like Glow or PixelCNN, which are computationally expensive and difficult to optimize. This paper proposes a more efficient alternative: using a standard binary classifier to perform implicit density ratio estimation.

Methodology

The authors propose three primary innovations to streamline OOD detection:

  1. Unified Energy-Based Framework: They demonstrate that existing density-ratio methods can be unified under a single framework where the ratio represents the unnormalized density of an implicit semantic distribution.
  2. Class Ratio Estimation: Leveraging a fundamental property of binary classifiers, the authors show that a model trained to distinguish between in-distribution ($p_{in}$) and base ($p_{base}$) samples implicitly estimates the density ratio. This allows for the extraction of OOD scores directly from a standard ResNet classifier's output.
  3. Spread Density Ratio: To prevent numerical instability (the "density chasm") where the classifier becomes too proficient at distinguishing distributions, the authors introduce small amounts of Gaussian noise to both distributions during training. This "spreads" the distributions, making the ratio estimation more robust.
Conceptual diagram of the density ratio framework showing in-distribution components within a broader base distribution.

Experimental Evaluation

The method was evaluated using standard benchmarks, including MNIST, FashionMNIST, CIFAR-10, and CIFAR-100 as in-distribution data, with SVHN, LSUN, and CelebA as OOD targets. The authors tested two types of base distributions: "Local" (augmented versions of the training data) and "Universal" (a diverse subset of the 80 Million Tiny Images dataset).

Key Results

  • State-of-the-Art Performance: On the CIFAR-10 vs. SVHN benchmark, the "Universal" base method achieved an AUROC of 98.2%, significantly outperforming complex generative methods such as Likelihood Ratio (91.2%) and Tiny-Glow (93.9%).
  • Computational Efficiency: By eliminating the need for generative models, the proposed method is significantly more parameter-efficient and easier to train than previous high-performing density ratio estimators.
  • Training Stability: The "Spread" technique proved essential for maintaining performance. As shown in the experimental results, adding noise prevents the AUROC fluctuations commonly seen in OOD detection training.
Stability of AUROC performance across various OOD benchmarks comparing the spread technique (w noise) against the baseline (w/o noise).

Conclusion

By reframing OOD detection as class ratio estimation, this work demonstrates that simple discriminative models can outperform or match the performance of far more complex generative architectures. The choice of a diverse "Universal" base distribution and the application of noise to stabilize training are key factors in the method's success. Future work may focus on constructing effective local base distributions without relying on external datasets and further refining the theoretical properties of the base distribution.

Summary updated

Abstract

Density-based Out-of-distribution (OOD) detection has recently been shown unreliable for the task of detecting OOD images. Various density ratio based approaches achieve good empirical performance, however methods typically lack a principled probabilistic modelling explanation. In this work, we propose to unify density ratio based methods under a novel framework that builds energy-based models and employs differing base distributions. Under our framework, the density ratio can be viewed as the unnormalized density of an implicit semantic distribution. Further, we propose to directly estimate the density ratio of a data sample through class ratio estimation. We report competitive results on OOD image problems in comparison with recent work that alternatively requires training of deep generative models for the task. Our approach enables a simple and yet effective path towards solving the OOD detection problem.
BibTeX
@misc{zhang2022out,
  author = {Mingtian Zhang and Andi Zhang and Tim Z. Xiao and Yitong Sun and Steven McDonagh},
  title = {{Out-of-Distribution Detection with Class Ratio Estimation}},
  year = {2022},
  eprint = {2206.03955},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.03955},
}
Figure