Abstract
Out-of-distribution (OOD) detection is critical for the safe deployment of deep learning models in real-world environments. While current state-of-the-art methods leverage internal network features to identify OOD samples, they often struggle with "hard" OOD cases where samples are semantically similar to the training distribution. This paper introduces a simple yet effective training framework that improves OOD detection by calibrating softmax probabilities through consistency regularization and self-supervised rotation tasks. Our approach significantly outperforms complex feature-based methods on challenging benchmarks while maintaining computational efficiency.
What Problem Does This Paper Address?
Deep Neural Networks (DNNs) are prone to overconfidence, often assigning high-probability labels to inputs they have never encountered. This behavior poses significant risks in safety-critical applications like autonomous driving or medical diagnostics.
Existing OOD detection techniques typically rely on distance metrics within the network's internal feature space (e.g., Mahalanobis distance). However, the authors observe a fundamental limitation: these methods are effective for "easy" OOD detection—where the OOD data is visually distinct (like digits vs. natural objects)—but fail on "hard" OOD detection, where the OOD data shares low-level features with the training set.
As shown in the figure above, while easy OOD samples like SVHN are clearly separated from the in-distribution CIFAR-10 data, hard OOD samples from CIFAR-100 overlap heavily in the feature space. In these difficult scenarios, complex state-of-the-art methods can actually perform worse than a simple softmax-based baseline.
Proposed Methodology
The framework focuses on making the model's final softmax output a more reliable indicator of distribution membership through three main components:
- Consistency Regularization: The model is trained on pairs of "weakly" and "strongly" augmented versions of the same image. By minimizing the Kullback-Leibler (KL) divergence between the resulting softmax distributions, the model is forced to focus on robust, high-level structural features rather than transient noise.
- Self-Supervised Auxiliary Task: The network is trained to predict the rotation angle (0°, 90°, 180°, or 270°) of input images. This auxiliary loss encourages the model to learn more comprehensive representations of the in-distribution data's geometry and structure.
- Augment-Entropy Inference: During testing, the model evaluates multiple augmented versions of a single input. The final OOD score is derived from the entropy of the averaged softmax probabilities across these augmentations. High entropy serves as a strong signal that the sample is out-of-distribution.
Experimental Setup and Datasets
The researchers benchmarked their framework against a variety of scenarios:
- In-Distribution: CIFAR-10, CIFAR-100, and the CUB-200 bird dataset.
- Easy OOD: SVHN, resized LSUN, and resized ImageNet.
- Hard OOD: CIFAR-100 (relative to CIFAR-10), fixed-resolution LSUN/ImageNet, and fine-grained datasets like Food-101.
- One-Class Remove: A specialized CIFAR-10 benchmark where one class is withheld during training to serve as the OOD target.
Key Insights
- The Internal Feature Limitation: Internal layers are often deceptive for hard OOD detection because they capture low-level textures and colors shared between similar datasets. High-level structural information, captured in the final layers, is more discriminative for hard cases.
- Calibration over Complexity: Improving the calibration of the model's own confidence scores is often more effective than designing complex external distance metrics.
- Computational Efficiency: The proposed method is highly efficient, requiring approximately 1/15th the training time of leading contrastive learning-based methods like CSI, while delivering competitive or superior performance.
Results and Performance
The framework demonstrates substantial improvements across the board:
- Hard OOD Accuracy: For a model trained on CIFAR-10, the AUROC for detecting CIFAR-100 samples improved from 86.36% (baseline) to 92.97%.
- Improved Calibration: The Expected Calibration Error (ECE) on CIFAR-10 was reduced from 3.64% to 1.51%, indicating that the model's confidence scores are much better aligned with its actual accuracy.
- Fine-Grained Success: On the CUB-200 dataset, the method achieved a mean AUROC of 97.19%, significantly outperforming the Gram matrix-based method which scored 76.71%.
Conclusion
This work highlights the importance of softmax calibration for robust OOD detection. By combining consistency regularization with self-supervised learning, the proposed framework provides a simple, efficient, and powerful solution for identifying both easy and hard out-of-distribution samples, paving the way for more reliable deep learning deployments.
Summary updated