A Unified Approach Towards Active Learning and Out-of-Distribution Detection

Sebastian Schmidt, Leonard Schenk, Leo Schwinn, Stephan GünnemannSubmission to ICLR 2025 · Withdrawn

Abstract

Active Learning (AL) and Out-of-Distribution (OOD) detection are traditionally treated as separate challenges in the machine learning lifecycle. However, both tasks fundamentally involve identifying samples that lie in ambiguous or unexplored regions of a model's feature space. This paper introduces SISOM (Simultaneous Informative Sampling and Outlier Mining), a unified framework that addresses both AL and OOD detection using a single set of metrics. By leveraging gradient-enhanced feature space distances, SISOM identifies informative samples for training and anomalous samples during deployment. SISOM achieves state-of-the-art performance on major benchmarks, including a first-place ranking in the OpenOOD near-OOD category.

Introduction: Bridging the Gap

In standard machine learning workflows, Active Learning (choosing which data to label) and Out-of-Distribution Detection (identifying "alien" data at runtime) are handled by separate systems. This separation introduces significant overhead, requiring different hyperparameters, auxiliary models, and potentially conflicting training schemes.

SISOM posits that these tasks are fundamentally linked. A sample that is "informative" for AL often sits near decision boundaries or in unexplored feature space regions—exactly where "near-OOD" samples tend to appear. By treating these as a unified problem, developers can streamline the model lifecycle.

The unified machine learning lifecycle: SISOM integrates Active Learning during the training phase and OOD detection during the operation phase.

The SISOM Framework

SISOM moves beyond simple uncertainty metrics like entropy, focusing instead on a robust analysis of the model's latent feature space. The framework consists of several key components:

  1. Neural Coverage: SISOM concatenates features from multiple network layers to provide a holistic view of how the model perceives an input.
  2. Feature Enhancement: To identify the most relevant neurons, SISOM weights features using the gradient of the KL divergence between the model's output and a uniform distribution. This "saliency weighting" highlights neurons responsible for the current decision.
  3. The Distance Ratio ($r$): For any new sample, SISOM calculates the ratio between the distance to the nearest sample of the predicted class ($d_{in}$) and the distance to the nearest sample of a different class ($d_{out}$). A high ratio indicates the sample is in a "no-man's land," making it a prime candidate for AL selection or OOD flagging.
  4. SISOMe (Self-Balancing Fusion): This variant automatically balances the distance ratio with an uncertainty-based Energy score, based on the average separability of the training data.
The SISOM Query process: (1) Coverage, (2) Feature Enhancement via gradients, (3) Distance Ratio calculation, and (4) Feature Space Analysis for AL and OOD tasks.

Experimental Evaluation

The authors evaluated SISOM across several standard benchmarks including CIFAR-10, CIFAR-100, SVHN, and ImageNet 1k. Standardized OOD benchmarking was conducted via the OpenOOD framework to compare against dozens of specialized methods.

Key Results

  • State-of-the-Art OOD Detection: SISOMe achieved 1st place overall in the combined ranking across CIFAR-10, CIFAR-100, and ImageNet for near-OOD data in the OpenOOD benchmark. It was the only method to maintain a top-3 ranking across all three datasets.
  • Competitive Active Learning: SISOM matched or outperformed specialized AL methods like CoreSet and Badge, particularly in high-data regimes where the model is relatively mature.
  • Lifecycle Synergy: "Full Life Cycle" experiments confirmed that models trained using SISOM for AL were inherently better at OOD detection once deployed.

Insights and Discussion

The Ambiguity Advantage: The research demonstrates that near-OOD data and informative AL samples overlap significantly in feature space. SISOM exploits this ambiguity rather than attempting to decouple it.

Efficiency via Subset Selection: To scale distance calculations, the authors used "Reduced Subset Selection" (RS), comparing new samples to only 5-10% of the labeled data. This not only improved speed but also enhanced OOD detection by filtering out noisy labeled outliers.

Limitations: Despite optimizations, calculating pairwise distances in high-dimensional space remains more computationally expensive than simple logit-based metrics. Additionally, while SISOM is effective for classification, future work is needed to extend this logic to tasks like object detection or semantic segmentation.

Conclusion

SISOM provides a robust, unified solution for two critical stages of the machine learning lifecycle. By focusing on the geometry of the feature space, it reduces system complexity while delivering top-tier performance in both data selection and anomaly detection.

Summary updated

Abstract

When applying deep learning models in real-world scenarios, active learning (AL) strategies are crucial for identifying label candidates from a nearly infinite amount of unlabeled data. In this context, robust out-of-distribution (OOD) detection mechanisms are essential for handling data outside the target distribution of the application. However, current works investigate both problems separately. In this work, we introduce SISOM as the first unified solution for both AL and OOD detection. By leveraging feature space distance metrics SISOM combines the strengths of the currently independent tasks to solve both effectively. We conduct extensive experiments showing the problems arising when migrating between both tasks. In these evaluations SISOM underlined its effectiveness by achieving first place in two of the widely used OpenOOD benchmarks and second place in the remaining one. In AL, SISOM outperforms others and delivers top-1 performance in three benchmarks.
BibTeX
@misc{schmidt2024unified,
  author = {Sebastian Schmidt and Leonard Schenk and Leo Schwinn and Stephan Günnemann},
  title = {{A Unified Approach Towards Active Learning and Out-of-Distribution Detection}},
  year = {2024},
  eprint = {2405.11337},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.11337},
}
Figure