Abstract
Out-of-Distribution (OOD) detection is essential in real-world applications, which has attracted increasing attention in recent years. However, most existing OOD detection methods require many labeled In-Distribution (ID) data, causing a heavy labeling cost. In this paper, we focus on the more realistic scenario, where limited labeled data and abundant unlabeled data are available, and these unlabeled data are mixed with ID and OOD samples. We propose the Adaptive In-Out-aware Learning (AIOL) method, in which we employ the appropriate temperature to adaptively select potential ID and OOD samples from the mixed unlabeled data and consider the entropy over them for OOD detection. Moreover, since the test data in realistic applications may contain OOD samples whose classes are not in the mixed unlabeled data (we call them unseen OOD classes), data augmentation techniques are brought into the method to further improve the performance. The experiments are conducted on various benchmark datasets, which demonstrate the superiority of our method.
Introduction
Deep learning models are notoriously overconfident, often making "confidently wrong" predictions when they encounter data from classes they were not trained on. This phenomenon poses significant risks in safety-critical applications, making Out-of-Distribution (OOD) detection a vital area of research. However, conventional OOD detection frameworks often rely on two idealistic assumptions: the availability of large-scale labeled In-Distribution (ID) datasets and the existence of "pure" unlabeled data (either entirely ID or entirely OOD).
In real-world scenarios, labeling is expensive, and unlabeled data is typically a "messy" mixture of ID samples and OOD samples. Furthermore, a model may encounter "seen" OOD classes (those present in the unlabeled training pool) and "unseen" OOD classes (entirely new categories encountered only during deployment). This paper introduces Adaptive In-Out-aware Learning (AIOL), a framework designed to bridge the gap between academic benchmarks and these complex, label-starved conditions.
Methodology
AIOL addresses the challenges of mixed unlabeled data through three primary mechanisms: adaptive temperature scaling, dynamic sample selection, and entropy optimization.
Adaptive Temperature Scaling
In semi-supervised learning, it is common to use a temperature parameter $T < 1$ to "sharpen" probability distributions. AIOL takes a counter-intuitive approach. The authors provide theoretical evidence (Theorem 1) demonstrating that using a temperature $T > 1$ actually increases the separation between the confidence scores of ID and OOD samples. By "softening" the distribution, the model can more effectively distinguish between the two. AIOL dynamically optimizes this temperature based on a validation set to maintain optimal discriminative power.
Dynamic GMM Selection
To handle the mixed unlabeled pool, AIOL employs a two-component Gaussian Mixture Model (GMM). In each training epoch, the model fits the GMM to the confidence scores of the unlabeled data. This allows the framework to dynamically partition the unlabeled pool into potential ID samples ($U^{in}$) and potential OOD samples ($U^{out}$), adapting as the model's feature representation improves.
Entropy Optimization and Augmentation
Once the samples are partitioned, the model applies dual entropy objectives:
- Entropy Minimization: Applied to predicted ID samples to enforce high-confidence predictions on known classes.
- Entropy Maximization: Applied to predicted OOD samples to force the model toward high uncertainty, preventing overconfident misclassifications.
To ensure the model generalizes to "unseen" OOD classes, AIOL incorporates RandAugment and a modified version of Mixup. This modification ensures that augmented samples remain semantically grounded, creating a "buffer" in the feature space that helps the model recognize novel OOD classes it never encountered during training.
Experimental Results
The researchers evaluated AIOL using standard benchmarks, including CIFAR-10 and CIFAR-100 as ID data, with ImageNet, SVHN, and various noise datasets (Blobs, Texture, iSUN) as OOD sources.
Performance Highlights
- High Detection Accuracy: On the CIFAR-10 vs. ImageNet benchmark, AIOL achieved an AUROC of 99.9% for seen OOD classes and 96.9% for unseen OOD classes.
- Label Efficiency: AIOL is remarkably robust to data scarcity. Even with only 250 labeled samples, the method maintained an AUROC of over 99% for seen OOD classes, significantly outperforming baselines like FixMatch and Outlier Exposure (OE), which saw substantial performance drops.
- Stability: AIOL remained consistent across various mixture ratios in the unlabeled pool, demonstrating its readiness for real-world data streams where the proportion of OOD data is unknown.
Discussion and Conclusion
A key insight from this work is the "Temperature Trick"—the finding that softening distributions ($T > 1$) is superior for OOD separation. While this provides a strong theoretical and empirical basis for OOD detection, the authors note a slight trade-off: the raw classification accuracy on ID samples can be marginally lower than pure semi-supervised methods like FixMatch. This occurs because the temperature setting optimized for OOD detection slightly blurs the sharp decision boundaries required for fine-grained classification.
In conclusion, AIOL provides a robust framework for OOD detection in realistic, low-label environments. By effectively exploiting mixed unlabeled data and utilizing adaptive scaling, it sets a new standard for identifying both seen and unseen OOD samples. Future work may extend this approach to complex domains such as medical imaging, where the cost of labeling is high and the presence of OOD anomalies is a critical concern.
Summary updated