Abstract
Many real-world ML deployments face the challenge of training a rare category model with a small labeling budget. In these settings, there is often access to large amounts of unlabeled data, therefore it is attractive to consider semi-supervised or active learning approaches to reduce human labeling effort. However, prior approaches make two assumptions that do not often hold in practice; (a) one has access to a modest amount of labeled data to bootstrap learning and (b) every image belongs to a common category of interest. In this paper, we consider the scenario where we start with as-little-as five labeled positives of a rare category and a large amount of unlabeled data of which 99.9% of it is negatives. We propose an active semi-supervised method for building accurate models in this challenging setting. Our method leverages two key ideas: (a) Utilize human and machine effort where they are most effective; human labels are used to identify "needle-in-a-haystack" positives, while machine-generated pseudo-labels are used to identify negatives. (b) Adapt recently proposed representation learning techniques for handling extremely imbalanced human labeled data to iteratively train models with noisy machine labeled data. We compare our approach with prior active learning and semi-supervised approaches, demonstrating significant improvements in accuracy per unit labeling effort, particularly on a tight labeling budget.
Introduction
In many practical machine learning scenarios—such as identifying rare bird species or specific construction vehicles in autonomous driving logs—the categories of interest are extremely scarce. Standard approaches to model training often fail in these "rare category" regimes due to two primary hurdles: the cold start problem and extreme data imbalance.
Most active learning methods require hundreds of initial labeled examples to bootstrap a model. When only a handful of examples (e.g., five) are available, the model lacks the discriminative power to intelligently query a human for labels. Furthermore, in massive unlabeled datasets, the vast majority (often >99.9%) of images are "background" noise. Random sampling for human annotation is inefficient, and standard semi-supervised techniques often fail by incorrectly identifying background images as positives.
Proposed Framework
To address these challenges, we propose an iterative, human-in-the-loop system that strategically divides labor between humans and machines.
Strategic Division of Labor
The system employs a "dual-labeling" strategy. Humans are tasked with labeling "hard" examples—those the model ranks as likely positives or those near the decision boundary. Simultaneously, the machine automatically "pseudo-labels" the bottom 50% of ranked images as easy negatives. This provides the model with a massive volume of negative data without consuming human labeling budget.
Background Splitting and Feature Evolution
To prevent the model from being overwhelmed by the extreme imbalance, we utilize "background splitting." This technique treats negatives as a diverse set of auxiliary classes rather than a single monolithic "negative" class, facilitating better feature learning. Unlike prior work that uses frozen features, our method fine-tunes the deep feature representation specifically for the rare category, which we found to be a significant performance driver.
Heterogeneous Model Updates
To maintain an interactive system, we update a simple linear model on cached features frequently (every 10 labels) to select the next human queries. The more computationally expensive deep feature representation is updated less frequently (every 50-100 labels).
Evaluation and Results
We evaluated our approach on the iNaturalist and Places datasets, specifically selecting rare categories that represent only ~0.01% to 0.27% of the total data. Our results demonstrate that the proposed method is highly efficient, matching the accuracy of a fully supervised model on iNaturalist using only 500 binary labels per category. It significantly outperforms existing baselines like Tropel; for instance, with a budget of 200 labels on iNaturalist, our method achieved an F1 score of ~35 compared to ~20 for Tropel.
Key insights from our experiments include:
- Negative Pseudo-labeling: While pseudo-labeling positives is often detrimental due to high error rates in rare categories, pseudo-labeling negatives is highly effective.
- Adaptive Sampling: The system's ability to switch from "likely positive" sampling to "uncertainty" sampling as it finds more positives is crucial for refining the decision boundary.
Conclusion
This work provides a robust framework for building rare category classifiers on a tight budget. While the periodic retraining of deep backbones presents a computational challenge, the strategic use of human effort for positive identification and machine effort for negative identification proves to be a powerful paradigm. Future work will explore scaling this to thousands of categories simultaneously and reducing the dependence on supervised pre-training.
Summary updated