Can We Ignore Labels in Out of Distribution Detection?

Hong Yang, Qi Yu, Travis Desell

Abstract

Out-of-distribution (OOD) detection is a critical component for the deployment of safe machine learning systems. While recent research has pivoted toward unlabeled OOD detection—leveraging self-supervised learning (SSL) to avoid the costs of manual labeling—this paper identifies a fundamental theoretical flaw termed "label blindness." We demonstrate that models trained without label information are mathematically guaranteed to discard semantic features necessary for distinguishing OOD samples that share visual similarities with the training distribution. Through the introduction of an "Adjacent OOD" benchmark, we show that popular unlabeled methods fail significantly in realistic scenarios, even when they appear successful on standard "Far OOD" benchmarks.

Introduction

In safety-critical domains such as autonomous driving and medical diagnostics, a model's ability to identify inputs that fall outside its training distribution is paramount. Traditionally, OOD detection has relied on supervised learning. However, the high cost of labeling has led to a surge in unlabeled OOD detection methods, which utilize self-supervised learning (SSL) or generative modeling to identify anomalies.

This paper argues that the research community’s reliance on standard OOD benchmarks has created a false sense of security. Most benchmarks focus on "Far OOD" tasks—for instance, distinguishing between handwritten digits and natural images. In these cases, low-level visual features are sufficient for detection. However, in real-world "Adjacent OOD" scenarios, where an OOD sample might share significant visual features with the training data but belong to a different semantic category, unlabeled methods suffer from a catastrophic failure known as label blindness.

The Label Blindness Theorem

The authors provide a rigorous mathematical foundation for this failure using the Information Bottleneck framework. The Label Blindness Theorem posits that if a model’s training objective (e.g., predicting image rotation or contrastive instance discrimination) is independent of the actual semantic labels, the model will naturally optimize for a "minimal" representation. In doing so, it discards information that is irrelevant to the surrogate task but critical for defining class boundaries. Consequently, the model becomes "blind" to any OOD shifts that occur within the label space rather than the feature space.

Adjacent OOD: A New Benchmarking Standard

To expose the limitations of unlabeled methods, the authors propose the "Adjacent OOD" detection task. Unlike traditional benchmarks that compare disparate datasets, Adjacent OOD involves splitting a single dataset by its semantic classes. For example, a model might be trained on a subset of vehicle types and then tasked with identifying other vehicle types as OOD. This requires the model to rely on fine-grained semantic understanding rather than simple texture or color differences.

Evaluated Methods and Datasets

The study benchmarks several prominent unlabeled approaches:

  • SimCLR: A contrastive learning framework.
  • Rotation Loss: A pretext-based SSL method.
  • Diffusion-based Models: Generative approaches used for anomaly detection.

These methods were tested against a supervised baseline across several challenging datasets:

  • ICML Facial Expressions: Training on specific emotions (e.g., happy, sad) and detecting others (e.g., angry) as OOD.
  • Stanford Cars & Food 101: Distinguishing between visually similar makes of cars or types of cuisine.
  • CIFAR-10 & CIFAR-100: Used to contrast performance between Far OOD and Adjacent OOD tasks.

Key Results and Performance Collapse

The quantitative findings reveal a stark disparity between supervised and unlabeled performance in adjacent scenarios:

  1. The Performance Gap: On the ICML Facial Expressions dataset, the supervised baseline achieved an AUROC of 70.8%. In contrast, SimCLR plummeted to 52.0%, performing only marginally better than a random guess (50%).
  2. The Illusion of Success: The same unlabeled methods that failed the Adjacent OOD tests achieved near-perfect scores (~99% AUROC) on Far OOD tasks (e.g., CIFAR vs. SVHN). This confirms that high performance on current benchmarks does not guarantee safety in more nuanced, real-world environments.
  3. Zero-Shot Limitations: Even large-scale models like CLIP exhibit label blindness. If the pretraining data lacks a strong semantic link between visual features and the specific labels of a downstream task, the model fails to detect OOD samples effectively.

Discussion and Future Directions

The findings suggest that the current trajectory of purely unlabeled OOD detection may be insufficient for safety-critical applications. The authors advocate for a shift in focus toward few-shot OOD detection, where a small amount of labeled data is used to anchor the model’s representations to relevant semantic categories.

Furthermore, the paper calls for the adoption of Adjacent OOD benchmarks as a standard requirement for evaluating new algorithms. By moving beyond "easy" Far OOD tasks, the community can develop models that are truly robust to the subtle semantic shifts encountered in the real world. While labels may be expensive, this research suggests they are a necessary ingredient for reliable out-of-distribution detection.

Summary updated

Abstract

Out-of-distribution (OOD) detection methods have recently become more prominent, serving as a core element in safety-critical autonomous systems. One major purpose of OOD detection is to reject invalid inputs that could lead to unpredictable errors and compromise safety. Due to the cost of labeled data, recent works have investigated the feasibility of self-supervised learning (SSL) OOD detection, unlabled OOD detection, and zero shot OOD detection. In this work, we identify a set of conditions for a theoretical guarantee of failure in unlabeled OOD detection algorithms from an information-theoretic perspective. These conditions are present in all OOD tasks dealing with real world data: I) we provide theoretical proof of unlabeled OOD detection failure when there exists zero mutual information between the learning objective and the in-distribution labels, a.k.a. ‘label blindness’, II) we define a new OOD task – Adjacent OOD detection – that tests for label blindness and accounts for a previously ignored safety gap in all OOD detection benchmarks, and III) we perform experiments demonstrating that existing unlabeled OOD methods fail under conditions suggested by our label blindness theory and analyze the implications for future research in unlabeled OOD methods.
BibTeX
@inproceedings{yang2025can,
  author = {Hong Yang and Qi Yu and Travis Desell},
  title = {{Can We Ignore Labels in Out of Distribution Detection?}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=falBlwUsIH},
}
Figure