DOI: Divergence-based Out-of-Distribution Indicators via Deep Generative Models

Wenxiao Chen, Xiaohui Nie, Mingliang Li, Dan PeiarXiv 2021 · arXiv

Introduction

Deep generative models (DGMs) are powerful tools for learning complex data distributions. However, a significant hurdle in their deployment for safety-critical applications is their tendency to assign higher likelihoods to out-of-distribution (OoD) samples than to the data they were trained on. This "likelihood paradox" undermines the reliability of using raw probability scores for anomaly detection. This paper introduces a novel framework, Divergence-based Out-of-Distribution Indicators (DOI), and a practical Single-shot Fine-tune (SF) algorithm to address this fundamental flaw.

The Likelihood Paradox

In a typical scenario, one would expect a model trained on a specific dataset (e.g., CIFAR-10) to assign the highest probabilities to similar images and lower probabilities to vastly different ones (e.g., SVHN house numbers). In practice, the opposite often occurs.

The Likelihood Paradox: A model trained on CIFAR-10 (red) assigns higher log-likelihoods to SVHN (green) test data.

As shown in the figure above, the distribution of log-likelihoods for the out-of-distribution SVHN dataset is shifted to the right of the in-distribution CIFAR-10 training set. This phenomenon suggests that static likelihood is an unreliable metric for OoD detection.

Methodology: Single-shot Fine-tuning

The authors propose moving away from static likelihoods toward measuring the "learning potential" of a sample. The core idea is that a model will adapt much more rapidly to an unseen OoD sample than to a sample from its known training distribution.

The Single-shot Fine-tune (SF) algorithm implements this by:

  1. Taking a single test image $x$.
  2. Fine-tuning the model on that specific image (using data augmentations like rotations to create a mini-batch) for exactly 64 steps.
  3. Comparing the difference: If the model’s likelihood for that image improves significantly after this brief "study session," the image is likely Out-of-Distribution. If the likelihood barely changes, the model already "knew" the distribution.

This approach is inductive, processing each sample independently, which is crucial for real-world applications where large batches of test data are not available.

Large-Scale Benchmarking

To ensure the robustness of their findings, the authors developed a massive benchmark consisting of 92 dataset pairs. This is a significant expansion over previous studies that typically relied on a handful of pairs. The benchmark includes a diverse range of data, from simple digits (MNIST, Omniglot) to complex natural images (CIFAR, CelebA) and synthetic noise. This extensive testing revealed that many existing OoD detection methods are "fragile" and fail when moved beyond the specific dataset pairs they were originally tuned for.

Key Results

The SF method demonstrates remarkable consistency and accuracy across various model architectures, including Variational Autoencoders (VAE), PixelCNN, and RNVP.

Distribution of the DOI indicator across multiple datasets, showing clear separation and a high AUROC of 0.9867.

The SF algorithm consistently outperformed state-of-the-art methods by 5–8% in AUROC. On the 92-pair benchmark, it achieved average AUROCs between 94.34% and 97.64%. The figure above illustrates how effectively the DOI framework can separate various types of data, including noise and different image domains, which were previously indistinguishable using raw likelihood.

Discussion and Limitations

While highly effective, the SF method introduces computational overhead, taking approximately 7 seconds per image on a standard GPU. This is significantly slower than a single forward pass but remains feasible for many high-stakes applications where accuracy is paramount. Furthermore, the method's success is tied to the quality of data augmentations used during the fine-tuning step to prevent simple memorization.

Conclusion

The DOI framework shifts the paradigm of OoD detection from "what the model knows" to "how much the model can learn." By leveraging the rapid adaptation of deep generative models to unseen data, the Single-shot Fine-tune algorithm provides a robust and generalizable solution to the likelihood paradox, validated across the most extensive OoD benchmark to date.

Summary updated

Abstract

To ensure robust and reliable classification results, OoD (out-of-distribution) indicators based on deep generative models are proposed recently and are shown to work well on small datasets. In this paper, we conduct the first large collection of benchmarks (containing 92 dataset pairs, which is 1 order of magnitude larger than previous ones) for existing OoD indicators and observe that none perform well. We thus advocate that a large collection of benchmarks is mandatory for evaluating OoD indicators. We propose a novel theoretical framework, DOI, for divergence-based Out-of-Distribution indicators (instead of traditional likelihood-based) in deep generative models. Following this framework, we further propose a simple and effective OoD detection algorithm: Single-shot Fine-tune. It significantly outperforms past works by 5~8 in AUROC, and its performance is close to optimal. In recent, the likelihood criterion is shown to be ineffective in detecting OoD. Single-shot Fine-tune proposes a novel fine-tune criterion to detect OoD, by whether the likelihood of the testing sample is improved after fine-tuning a well-trained model on it. Fine-tune criterion is a clear and easy-following criterion, which will lead the OoD domain into a new stage.
BibTeX
@misc{chen2021doi,
  author = {Wenxiao Chen and Xiaohui Nie and Mingliang Li and Dan Pei},
  title = {{DOI: Divergence-based Out-of-Distribution Indicators via Deep Generative Models}},
  year = {2021},
  eprint = {2108.05509},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.05509},
}
Figure