Abstract
Active Learning (AL) and Out-of-Distribution (OOD) detection are traditionally treated as separate challenges in the machine learning lifecycle. However, both tasks fundamentally involve identifying samples that lie in ambiguous or unexplored regions of a model's feature space. This paper introduces SISOM (Simultaneous Informative Sampling and Outlier Mining), a unified framework that addresses both AL and OOD detection using a single set of metrics. By leveraging gradient-enhanced feature space distances, SISOM identifies informative samples for training and anomalous samples during deployment. SISOM achieves state-of-the-art performance on major benchmarks, including a first-place ranking in the OpenOOD near-OOD category.
Introduction: Bridging the Gap
In standard machine learning workflows, Active Learning (choosing which data to label) and Out-of-Distribution Detection (identifying "alien" data at runtime) are handled by separate systems. This separation introduces significant overhead, requiring different hyperparameters, auxiliary models, and potentially conflicting training schemes.
SISOM posits that these tasks are fundamentally linked. A sample that is "informative" for AL often sits near decision boundaries or in unexplored feature space regions—exactly where "near-OOD" samples tend to appear. By treating these as a unified problem, developers can streamline the model lifecycle.
The SISOM Framework
SISOM moves beyond simple uncertainty metrics like entropy, focusing instead on a robust analysis of the model's latent feature space. The framework consists of several key components:
- Neural Coverage: SISOM concatenates features from multiple network layers to provide a holistic view of how the model perceives an input.
- Feature Enhancement: To identify the most relevant neurons, SISOM weights features using the gradient of the KL divergence between the model's output and a uniform distribution. This "saliency weighting" highlights neurons responsible for the current decision.
- The Distance Ratio ($r$): For any new sample, SISOM calculates the ratio between the distance to the nearest sample of the predicted class ($d_{in}$) and the distance to the nearest sample of a different class ($d_{out}$). A high ratio indicates the sample is in a "no-man's land," making it a prime candidate for AL selection or OOD flagging.
- SISOMe (Self-Balancing Fusion): This variant automatically balances the distance ratio with an uncertainty-based Energy score, based on the average separability of the training data.
Experimental Evaluation
The authors evaluated SISOM across several standard benchmarks including CIFAR-10, CIFAR-100, SVHN, and ImageNet 1k. Standardized OOD benchmarking was conducted via the OpenOOD framework to compare against dozens of specialized methods.
Key Results
- State-of-the-Art OOD Detection: SISOMe achieved 1st place overall in the combined ranking across CIFAR-10, CIFAR-100, and ImageNet for near-OOD data in the OpenOOD benchmark. It was the only method to maintain a top-3 ranking across all three datasets.
- Competitive Active Learning: SISOM matched or outperformed specialized AL methods like CoreSet and Badge, particularly in high-data regimes where the model is relatively mature.
- Lifecycle Synergy: "Full Life Cycle" experiments confirmed that models trained using SISOM for AL were inherently better at OOD detection once deployed.
Insights and Discussion
The Ambiguity Advantage: The research demonstrates that near-OOD data and informative AL samples overlap significantly in feature space. SISOM exploits this ambiguity rather than attempting to decouple it.
Efficiency via Subset Selection: To scale distance calculations, the authors used "Reduced Subset Selection" (RS), comparing new samples to only 5-10% of the labeled data. This not only improved speed but also enhanced OOD detection by filtering out noisy labeled outliers.
Limitations: Despite optimizations, calculating pairwise distances in high-dimensional space remains more computationally expensive than simple logit-based metrics. Additionally, while SISOM is effective for classification, future work is needed to extend this logic to tasks like object detection or semantic segmentation.
Conclusion
SISOM provides a robust, unified solution for two critical stages of the machine learning lifecycle. By focusing on the geometry of the feature space, it reduces system complexity while delivering top-tier performance in both data selection and anomaly detection.
Summary updated