Entropic Open-Set Active Learning

Bardia Safaei, Vibashan Vs, Celso M. de Melo, Vishal M. Patel

Introduction

Active Learning (AL) aims to train high-performing deep models with minimal labeled data by strategically selecting the most informative samples for human annotation. However, traditional AL algorithms typically operate under a "closed-set" assumption, where all unlabeled data is presumed to belong to known classes. In real-world scenarios, unlabeled pools are often contaminated with "unknown" classes or distractors. Standard uncertainty-based methods are frequently misled by these unknowns, wasting annotation budgets on samples that do not contribute to the primary task.

The EOAL Framework

To address this, we present Entropic Open-set AL (EOAL), a framework designed to distinguish between informative known samples and irrelevant unknowns. EOAL employs a "push-pull" mechanism utilizing two distinct entropy measures to refine the selection process.

The EOAL architecture illustrating the flow of known and unknown data through binary classifiers and clustering for query strategy.

Dual Entropy Mechanism

The core of EOAL lies in its query strategy, which combines:

  1. Closed-set Entropy ($S_c$): Utilizing $K$ binary classifiers (one per known class), this metric assesses uncertainty relative to known categories. While known samples ideally exhibit low entropy, unknown samples typically show high entropy due to model confusion.
  2. Distance-based Entropy ($S_d$): This novel component leverages "active unknowns"—samples identified as unknown in previous cycles. By clustering these unknowns, $S_d$ measures the proximity of new unlabeled samples to established unknown distributions.

The final selection score is calculated as $S(x) = S_c(x) - S_d(x)$. This approach prioritizes samples that are uncertain regarding known classes but distant from known unknown clusters. To ensure a diverse selection, the framework incorporates the FINCH clustering algorithm to sample across the unlabeled pool.

Experimental Evaluation

EOAL was evaluated on CIFAR-10, CIFAR-100, and TinyImageNet benchmarks. To simulate real-world conditions, "mismatch ratios" of 20%, 30%, and 40% were introduced, where a portion of the classes were treated as unknowns.

Key Results

EOAL consistently outperformed state-of-the-art (SOTA) methods such as LfOSA and MQNet across all tested scenarios.

Accuracy vs. AL Cycle across CIFAR-10, CIFAR-100, and TinyImageNet with a 20% mismatch ratio, showing EOAL's superior performance.

On CIFAR-100 with a 20% mismatch, EOAL achieved a classification accuracy 3.88% higher than MQNet and 6.00% higher than LfOSA. Notably, the performance gap widened in more challenging environments (e.g., 40% mismatch), demonstrating the robustness of the dual-entropy approach. Furthermore, EOAL maintained higher precision in selecting "known" samples compared to competitors, which often experienced precision degradation as cycles progressed.

Conclusion and Future Directions

EOAL demonstrates that discovered unknowns are not merely noise but valuable data for modeling the unknown distribution. By balancing purity and informativeness, EOAL effectively navigates the complexities of open-set active learning.

Future work may address the "cold start" problem for distance-based entropy in the initial AL round and explore adaptive clustering techniques to handle varying numbers of unknown categories. While the use of $K$ binary classifiers introduces some computational overhead, the significant gains in accuracy and annotation efficiency justify the complexity in mismatched real-world settings.

Summary updated

Abstract

Active Learning (AL) aims to enhance the performance of deep models by selecting the most informative samples for annotation from a pool of unlabeled data. Despite impressive performance in closed-set settings, most AL methods fail in real-world scenarios where the unlabeled data contains unknown categories. Recently, a few studies have attempted to tackle the AL problem for the open-set setting. However, these methods focus more on selecting known samples and do not efficiently utilize unknown samples obtained during AL rounds. In this work, we propose an Entropic Open-set AL (EOAL) framework which leverages both known and unknown distributions effectively to select informative samples during AL rounds. Specifically, our approach employs two different entropy scores. One measures the uncertainty of a sample with respect to the known-class distributions. The other measures the uncertainty of the sample with respect to the unknown-class distributions. By utilizing these two entropy scores we effectively separate the known and unknown samples from the unlabeled data resulting in better sampling. Through extensive experiments, we show that the proposed method outperforms existing state-of-the-art methods on CIFAR-10, CIFAR-100, and TinyImageNet datasets. Code is available at \url{https://github.com/bardisafa/EOAL}.
BibTeX
@inproceedings{safaei2024entropic,
  author = {Bardia Safaei and Vibashan VS and Celso M. de Melo and Vishal M. Patel},
  title = {{Entropic Open-Set Active Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i5.28269},
}
Figure