Abstract
Active Learning (AL) aims to minimize labeling costs by selecting the most informative samples. However, real-world unlabeled pools often contain "unknown" classes (open-set noise), which standard AL methods frequently misidentify as valuable. This paper introduces Dirichlet-Based Coarse-to-Fine Example Selection (DCFS), a framework that utilizes Dirichlet-based evidential learning and dual-classifier discrepancy to filter out unknown categories. By decoupling data and distribution uncertainty, DCFS selects high-value "known" examples, significantly outperforming existing methods on benchmarks like CIFAR and Tiny-ImageNet under various openness ratios.
Introduction: The Open-Set Challenge
Traditional Active Learning algorithms operate under a "closed-set" assumption, where every unlabeled image belongs to a predefined category. In practice, data sourced from the web often contains "unknown" classes. Standard AL methods typically fail in these scenarios: uncertainty-based methods select unknown classes due to model confusion, while diversity-based methods select them because they appear novel.
A primary technical hurdle is the softmax function. Due to its translation invariance, softmax focuses on relative differences between scores rather than absolute magnitudes. Consequently, an unknown class (e.g., a Lynx) can yield the same probability distribution as a known class (e.g., a Cat), making them indistinguishable to the selection algorithm.
Methodology: Dirichlet-Based Coarse-to-Fine Selection (DCFS)
To overcome the limitations of softmax, the authors propose DCFS, which transitions from point estimates to distribution-based evidential learning.
1. Simplex-Based Evidential Deep Learning (EDL)
Instead of a single probability vector, DCFS treats predictions as a Dirichlet distribution. This allows the model to accumulate "evidence" for each class. On the probability simplex, known classes typically show evidence concentrated at the corners, whereas unknown classes gravitate toward the center. This approach effectively breaks the translation invariance of softmax by distinguishing between "conflicting evidence" and a "lack of evidence."
2. Model Discrepancy and Two-Stage Selection
DCFS employs a dual-classifier architecture. During training, the discrepancy between these two heads is amplified for unknown classes and minimized for known ones. The selection process then follows a Coarse-to-Fine strategy:
- Coarse Stage: A Gaussian Mixture Model (GMM) combines model discrepancy and data uncertainty to filter out likely unknown-class examples.
- Fine Stage: From the filtered pool, the algorithm selects examples that maximize both data uncertainty (identifying "hard" known cases) and distribution uncertainty (identifying cases with low total evidence).
Key Results and Insights
The researchers evaluated DCFS on CIFAR10, CIFAR100, and Tiny-ImageNet with varying "openness ratios" (0.2 to 0.6).
- Superior Performance: DCFS consistently outperformed state-of-the-art (SOTA) methods. For instance, on CIFAR100 with a 0.4 openness ratio, DCFS achieved 59.9% accuracy, surpassing the next best method (CCAL) by 3.8%.
- Robustness to Noise: The performance gap between DCFS and competitors widened as the openness ratio increased, demonstrating its resilience in highly noisy environments.
- Effective Selection: t-SNE visualizations revealed that DCFS successfully avoids redundant examples, instead picking "hard" known samples that are distant from existing labeled clusters, thereby enhancing model generalization.
Conclusion and Future Work
DCFS addresses a critical flaw in Open-Set Annotation by decoupling uncertainty into data and distribution components. By moving beyond softmax to a Dirichlet-based framework, it provides a robust mechanism for filtering noise and selecting informative data. Future research may focus on reducing the computational overhead of dual-classifier training and exploring the method's scalability to large-scale transformer architectures like Vision Transformers (ViTs).
Summary updated