Exploiting Mixed Unlabeled Data for Detecting Samples of Seen and Unseen Out-of-Distribution Classes

Yi-Xuan Sun, Wei Wang

Abstract

Deep Neural Networks (DNNs) often exhibit overconfidence when encountering Out-of-Distribution (OOD) data, posing significant risks in safety-critical applications. Traditional OOD detection methods typically require large labeled datasets or assume "pure" unlabeled data. This paper presents Adaptive In-Out-aware Learning (AIOL), a framework designed for realistic scenarios where unlabeled data is a mixture of In-Distribution (ID) and OOD samples. AIOL utilizes adaptive temperature scaling and a Gaussian Mixture Model (GMM) for dynamic sample selection, enabling the model to effectively distinguish between known classes, seen OOD classes from the unlabeled pool, and entirely novel unseen OOD classes. Experimental results on CIFAR benchmarks demonstrate that AIOL significantly outperforms existing baselines, particularly in low-label regimes.

Introduction

The deployment of machine learning models in the real world necessitates robust Out-of-Distribution (OOD) detection to prevent confident but incorrect predictions on unfamiliar inputs. However, current research often relies on two impractical assumptions: the availability of massive labeled In-Distribution (ID) data and the purity of unlabeled datasets. In practice, unlabeled data harvested from the wild is often a "mixed bag" containing both ID and OOD samples. Furthermore, models must generalize to "unseen" OOD classes that were not present in the training mixture.

Examples of In-Distribution (ID) classes used during training.

AIOL addresses these challenges by treating OOD samples within the unlabeled pool as a valuable training signal rather than noise. By learning what "not-ID" looks like from the mixed data, the model becomes better equipped to identify novel classes during deployment.

Examples of Unseen Out-of-Distribution (OOD) classes encountered at test time.

Methodology: Adaptive In-Out-aware Learning (AIOL)

The AIOL framework is built upon three core components designed to maximize the utility of mixed unlabeled data.

Adaptive Temperature Scaling

A key theoretical contribution of this work is the justification for temperature scaling. While sharpening ($T < 1$) is common in semi-supervised learning to produce hard labels, the authors prove that softening ($T > 1$) in the softmax layer increases the confidence gap between ID and OOD samples. AIOL adaptively optimizes this temperature $T$ during training based on a small validation set to maintain optimal separation.

Dynamic Sample Selection via GMM

To handle the evolving nature of the model's confidence, AIOL avoids static thresholds. Instead, it fits a two-component Gaussian Mixture Model (GMM) to the confidence scores of the unlabeled data at every epoch. This allows the model to dynamically "self-label" unlabeled samples as either potential ID or OOD, adapting to the model's current state of learning.

Entropy Optimization and Augmentation

Once samples are categorized by the GMM:

  • ID Samples: The model undergoes entropy minimization to reinforce high-confidence predictions on known classes.
  • OOD Samples: The model performs entropy maximization, encouraging the network to express high uncertainty.

To improve generalization to unseen OOD classes, AIOL employs RandAugment and a modified Mixup strategy, increasing the diversity of the training signal.

Experimental Evaluation

The researchers evaluated AIOL using CIFAR-10 and CIFAR-100 as ID datasets, with various datasets (SVHN, ImageNet, etc.) serving as seen and unseen OOD benchmarks.

Key Results

  • Superior Detection: On CIFAR-10 vs. SVHN, AIOL achieved a near-perfect AUROC of 100.0%, significantly higher than the 54.2% achieved by standard baselines.
  • Generalization to Unseen Classes: AIOL maintained high performance even on datasets never encountered during training, such as a 96.9% AUROC on ImageNet as an unseen OOD.
  • Label Efficiency: The method is highly effective in low-data scenarios. With only 250 labeled samples, AIOL achieved 99.8% AUROC on seen OOD tasks, whereas methods like FixMatch dropped to 42.1%.

Conclusion

AIOL provides a robust framework for OOD detection by embracing the complexity of real-world unlabeled data. By leveraging adaptive temperature scaling and dynamic sample selection, it effectively bridges the gap between semi-supervised learning and OOD detection. While there is a slight trade-off in ID classification accuracy compared to pure semi-supervised methods, the gains in safety and reliability through superior OOD detection make AIOL a compelling approach for real-world AI deployment.

Summary updated

Abstract

Out-of-Distribution (OOD) detection is essential in real-world applications, which has attracted increasing attention in recent years. However, most existing OOD detection methods require many labeled In-Distribution (ID) data, causing a heavy labeling cost. In this paper, we focus on the more realistic scenario, where limited labeled data and abundant unlabeled data are available, and these unlabeled data are mixed with ID and OOD samples. We propose the Adaptive In-Out-aware Learning (AIOL) method, in which we employ the appropriate temperature to adaptively select potential ID and OOD samples from the mixed unlabeled data and consider the entropy over them for OOD detection. Moreover, since the test data in realistic applications may contain OOD samples whose classes are not in the mixed unlabeled data (we call them unseen OOD classes), data augmentation techniques are brought into the method to further improve the performance. The experiments are conducted on various benchmark datasets, which demonstrate the superiority of our method.
BibTeX
@inproceedings{sun2022exploiting,
  author = {Yi-Xuan Sun and Wei Wang},
  title = {{Exploiting Mixed Unlabeled Data for Detecting Samples of Seen and Unseen Out-of-Distribution Classes}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=8TWup6mM6c},
}
Figure