Introduction
Active Learning (AL) aims to train high-performing deep models with minimal labeled data by strategically selecting the most informative samples for human annotation. However, traditional AL algorithms typically operate under a "closed-set" assumption, where all unlabeled data is presumed to belong to known classes. In real-world scenarios, unlabeled pools are often contaminated with "unknown" classes or distractors. Standard uncertainty-based methods are frequently misled by these unknowns, wasting annotation budgets on samples that do not contribute to the primary task.
The EOAL Framework
To address this, we present Entropic Open-set AL (EOAL), a framework designed to distinguish between informative known samples and irrelevant unknowns. EOAL employs a "push-pull" mechanism utilizing two distinct entropy measures to refine the selection process.
Dual Entropy Mechanism
The core of EOAL lies in its query strategy, which combines:
- Closed-set Entropy ($S_c$): Utilizing $K$ binary classifiers (one per known class), this metric assesses uncertainty relative to known categories. While known samples ideally exhibit low entropy, unknown samples typically show high entropy due to model confusion.
- Distance-based Entropy ($S_d$): This novel component leverages "active unknowns"—samples identified as unknown in previous cycles. By clustering these unknowns, $S_d$ measures the proximity of new unlabeled samples to established unknown distributions.
The final selection score is calculated as $S(x) = S_c(x) - S_d(x)$. This approach prioritizes samples that are uncertain regarding known classes but distant from known unknown clusters. To ensure a diverse selection, the framework incorporates the FINCH clustering algorithm to sample across the unlabeled pool.
Experimental Evaluation
EOAL was evaluated on CIFAR-10, CIFAR-100, and TinyImageNet benchmarks. To simulate real-world conditions, "mismatch ratios" of 20%, 30%, and 40% were introduced, where a portion of the classes were treated as unknowns.
Key Results
EOAL consistently outperformed state-of-the-art (SOTA) methods such as LfOSA and MQNet across all tested scenarios.
On CIFAR-100 with a 20% mismatch, EOAL achieved a classification accuracy 3.88% higher than MQNet and 6.00% higher than LfOSA. Notably, the performance gap widened in more challenging environments (e.g., 40% mismatch), demonstrating the robustness of the dual-entropy approach. Furthermore, EOAL maintained higher precision in selecting "known" samples compared to competitors, which often experienced precision degradation as cycles progressed.
Conclusion and Future Directions
EOAL demonstrates that discovered unknowns are not merely noise but valuable data for modeling the unknown distribution. By balancing purity and informativeness, EOAL effectively navigates the complexities of open-set active learning.
Future work may address the "cold start" problem for distance-based entropy in the initial AL round and explore adaptive clustering techniques to handle varying numbers of unknown categories. While the use of $K$ binary classifiers introduces some computational overhead, the significant gains in accuracy and annotation efficiency justify the complexity in mismatched real-world settings.
Summary updated