A Simple Framework for Robust Out-of-Distribution Detection

Youngbum Hur, Eunho Yang, Sung Ju HwangIEEE Access 2022 · OpenReview

Abstract

Out-of-distribution (OOD) detection is critical for the safe deployment of deep learning models in real-world environments. While current state-of-the-art methods leverage internal network features to identify OOD samples, they often struggle with "hard" OOD cases where samples are semantically similar to the training distribution. This paper introduces a simple yet effective training framework that improves OOD detection by calibrating softmax probabilities through consistency regularization and self-supervised rotation tasks. Our approach significantly outperforms complex feature-based methods on challenging benchmarks while maintaining computational efficiency.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are prone to overconfidence, often assigning high-probability labels to inputs they have never encountered. This behavior poses significant risks in safety-critical applications like autonomous driving or medical diagnostics.

Existing OOD detection techniques typically rely on distance metrics within the network's internal feature space (e.g., Mahalanobis distance). However, the authors observe a fundamental limitation: these methods are effective for "easy" OOD detection—where the OOD data is visually distinct (like digits vs. natural objects)—but fail on "hard" OOD detection, where the OOD data shares low-level features with the training set.

Comparison of easy (SVHN) and hard (CIFAR-100) OOD detection scenarios.

As shown in the figure above, while easy OOD samples like SVHN are clearly separated from the in-distribution CIFAR-10 data, hard OOD samples from CIFAR-100 overlap heavily in the feature space. In these difficult scenarios, complex state-of-the-art methods can actually perform worse than a simple softmax-based baseline.

Proposed Methodology

The framework focuses on making the model's final softmax output a more reliable indicator of distribution membership through three main components:

  1. Consistency Regularization: The model is trained on pairs of "weakly" and "strongly" augmented versions of the same image. By minimizing the Kullback-Leibler (KL) divergence between the resulting softmax distributions, the model is forced to focus on robust, high-level structural features rather than transient noise.
  2. Self-Supervised Auxiliary Task: The network is trained to predict the rotation angle (0°, 90°, 180°, or 270°) of input images. This auxiliary loss encourages the model to learn more comprehensive representations of the in-distribution data's geometry and structure.
  3. Augment-Entropy Inference: During testing, the model evaluates multiple augmented versions of a single input. The final OOD score is derived from the entropy of the averaged softmax probabilities across these augmentations. High entropy serves as a strong signal that the sample is out-of-distribution.

Experimental Setup and Datasets

The researchers benchmarked their framework against a variety of scenarios:

  • In-Distribution: CIFAR-10, CIFAR-100, and the CUB-200 bird dataset.
  • Easy OOD: SVHN, resized LSUN, and resized ImageNet.
  • Hard OOD: CIFAR-100 (relative to CIFAR-10), fixed-resolution LSUN/ImageNet, and fine-grained datasets like Food-101.
  • One-Class Remove: A specialized CIFAR-10 benchmark where one class is withheld during training to serve as the OOD target.

Key Insights

  • The Internal Feature Limitation: Internal layers are often deceptive for hard OOD detection because they capture low-level textures and colors shared between similar datasets. High-level structural information, captured in the final layers, is more discriminative for hard cases.
  • Calibration over Complexity: Improving the calibration of the model's own confidence scores is often more effective than designing complex external distance metrics.
  • Computational Efficiency: The proposed method is highly efficient, requiring approximately 1/15th the training time of leading contrastive learning-based methods like CSI, while delivering competitive or superior performance.

Results and Performance

The framework demonstrates substantial improvements across the board:

  • Hard OOD Accuracy: For a model trained on CIFAR-10, the AUROC for detecting CIFAR-100 samples improved from 86.36% (baseline) to 92.97%.
  • Improved Calibration: The Expected Calibration Error (ECE) on CIFAR-10 was reduced from 3.64% to 1.51%, indicating that the model's confidence scores are much better aligned with its actual accuracy.
  • Fine-Grained Success: On the CUB-200 dataset, the method achieved a mean AUROC of 97.19%, significantly outperforming the Gram matrix-based method which scored 76.71%.

Conclusion

This work highlights the importance of softmax calibration for robust OOD detection. By combining consistency regularization with self-supervised learning, the proposed framework provides a simple, efficient, and powerful solution for identifying both easy and hard out-of-distribution samples, paving the way for more reliable deep learning deployments.

Summary updated

Abstract

Out-of-distribution (OOD) detection, i.e., identifying whether a given test sample is drawn from outside the training distribution, is essential for a deep classifier to be deployed in a real-world application. The existing state-of-the-art methods of OOD detection tackle this issue by utilizing the internal feature of the classification network. However, we found that such detection methods inherently struggle to detect hard OOD images, i.e., drawn near from the training distribution: a naive softmax-based baseline even outperforms them. Motivated by this, we propose a simple yet effective training scheme for further calibrating the softmax probability of a classifier to achieve high OOD detection performance under both hard and easy scenarios. In particular, we suggest to optimize consistency regularization and self-supervised loss during training. Our experiments demonstrate the superiority of our simple method under various OOD detection scenarios.
BibTeX
@inproceedings{hur2022simple,
  author = {Youngbum Hur and Eunho Yang and Sung Ju Hwang},
  title = {{A Simple Framework for Robust Out-of-Distribution Detection}},
  booktitle = {IEEE Access},
  year = {2022},
  url = {https://openreview.net/forum?id=yG6XpT6WlQ},
}
Figure