Abstract
In the lifecycle of deep learning models, two data-centric challenges often emerge as distinct engineering hurdles: Active Learning (AL) and Out-of-Distribution (OOD) detection. Traditionally, these tasks are addressed using separate frameworks and metrics. This paper introduces SISOM (Simultaneous Informative Sampling and Outlier Mining), a unified framework that treats AL and OOD detection as two sides of the same coin. By targeting the "ambiguous" regions between data clusters in latent space, SISOM achieves state-of-the-art performance in both identifying informative samples for training and detecting anomalous data during deployment.
Introduction
Machine learning developers face a dual challenge when moving from model development to real-world deployment. During the training phase, Active Learning is employed to identify which unlabeled samples are most valuable for human annotation, thereby reducing labeling costs. During the operational phase, OOD Detection is critical to ensure the model can identify and reject data it has never encountered before, preventing overconfident but incorrect predictions.
While these are typically treated as separate problems, SISOM is built on the Ambiguity Hypothesis: the most informative samples for training and the most dangerous samples for deployment often reside in the same "no man's land" between known class clusters in the model's feature space. SISOM exploits this overlap to provide a cohesive solution for the full model lifecycle.
The SISOM Framework
SISOM relies on three primary pillars to characterize data samples within the latent space:
1. Multi-Layer Coverage
Rather than relying solely on the final representation layer of a neural network, SISOM concatenates features from multiple layers. This creates a richer, high-dimensional "fingerprint" of the data, capturing both low-level structural details and high-level semantic information.
2. Feature Enhancement via Gradients
To identify which features are most relevant to the model's current state of uncertainty, SISOM uses the gradient of the KL divergence. By measuring how much a prediction differs from a uniform distribution (representing total uncertainty), the framework weights features based on their influence on the model's decision-making process. This effectively highlights the specific internal representations that are struggling with a given input.
3. The Distance Ratio ($r$)
The core metric of SISOM is the distance ratio ($r$). For any new sample, the framework calculates the ratio between the distance to the nearest sample of the same predicted class ($d_{in}$) and the distance to the nearest sample of a different class ($d_{out}$). A high ratio indicates that a sample is located near a decision boundary—the exact region where both informative AL samples and near-OOD samples are found.
SISOMe: Energy-Aware Fusion
The authors also introduce SISOMe, an "energy-aware" variant. SISOMe automatically balances the distance-based metric with model uncertainty (energy scores). This self-balancing mechanism is triggered by the current "separability" of the latent space ($r_{avg}$), allowing the method to remain robust even when the model's feature representations are still immature during early training stages.
Experimental Evaluation
The researchers evaluated SISOM across several standard benchmarks, including CIFAR-10, CIFAR-100, SVHN, and the comprehensive OpenOOD benchmark (including ImageNet-1k).
Out-of-Distribution Detection
SISOMe achieved 1st place overall in the combined ranking across CIFAR-10, CIFAR-100, and ImageNet-1k for near-OOD data in the OpenOOD benchmark. Notably, it was the only method to consistently rank in the top 3 across all three datasets, demonstrating superior generalization compared to specialized OOD baselines.
Active Learning Performance
In AL tasks, SISOM outperformed established baselines such as CoreSet and Badge. The method showed particularly strong "velocity," achieving rapid accuracy gains in the later stages of training where identifying subtle boundary cases becomes critical.
Full Life Cycle Integration
Experiments simulating the full model lifecycle revealed that models trained using SISOM's active learning strategy were inherently better at OOD detection during deployment. This suggests that the samples selected by SISOM help the model build more robust decision boundaries that naturally facilitate outlier rejection.
Efficiency and Scalability
A common bottleneck in distance-based methods is the computational cost of pairwise calculations. To address this, the authors introduced Reduced Subset Selection (RS). By using a representative 5-10% subset of the labeled data for comparison, SISOM maintains its performance while reducing runtime by approximately 400%.
Conclusion and Future Work
SISOM demonstrates that Active Learning and OOD detection are not only compatible but synergistic. By focusing on the ambiguous regions of the feature space, the framework provides a unified, efficient solution for data selection and anomaly detection. Future research will focus on extending SISOM to complex tasks like object detection and exploring "Open-set AL" scenarios, where the unlabeled pool may contain significant amounts of OOD noise.
Summary updated