TL;DR
This paper introduces AMEND, a novel framework for Generalized Category Discovery (GCD) that efficiently discovers and classifies both seen and unseen object categories in images. AMEND achieves state-of-the-art performance by leveraging expanded neighborhood information for robust feature learning and an adaptive margin regularizer to improve class separability, all while being computationally efficient.
What Problem Does This Paper Address?
In real-world scenarios, machine learning models often encounter data from classes they weren't explicitly trained on. Generalized Category Discovery (GCD) aims to tackle this by simultaneously classifying images from known (seen) classes and discovering/clustering images from unknown (unseen) classes. Existing GCD approaches often face a trade-off: highly accurate methods are computationally expensive, while efficient methods lag in performance. This paper addresses the need for a GCD framework that is both highly effective and computationally efficient, bridging this performance-efficiency gap.
Methods
The AMEND framework introduces two main innovations:
- Expanded Neighborhood for Robust Representations: Instead of just using an image and its augmentations as positive pairs in contrastive learning, AMEND incorporates information from both nearest neighbors and "expanded neighbors" (neighbors of neighbors). This richer neighborhood context helps learn more discriminative and generalizable feature representations, leading to better clustering.
- Adaptive Margin Regularizer: To address the issue of confusing classes, especially in fine-grained datasets, AMEND employs a class-wise adaptive margin regularizer. This regularizer dynamically adjusts the separation between class prototypes (representatives of each class) based on their current similarity. It aims to increase the angular separation between prototypes that are close to each other, thereby reducing confusion and improving classification accuracy.
The overall framework combines these components with a standard contrastive learning loss for representation learning and a cross-entropy loss for classification.
Datasets
The authors evaluated AMEND on a variety of datasets:
- Generic Datasets: CIFAR10, CIFAR100, and ImageNet-100.
- Fine-grained Datasets: Stanford Cars, CUB (Caltech-UCSD Birds), FGVC Aircraft, and Herbarium-19.
The Herbarium-19 dataset is noted for its long-tailed data distribution, presenting a significant challenge for class imbalance. The experiments used a standard split where a portion of classes and images were designated as seen/labeled, and the rest as unseen/unlabeled.
Key Insights and Ideas
- Neighborhood matters: Simply using augmentations isn't enough for robust feature learning in GCD. Incorporating information from neighboring data points, and even neighbors of neighbors, significantly boosts performance by capturing richer semantic context.
- Class confusion is a major hurdle: Fine-grained datasets and classes with similar visual characteristics pose a significant challenge. An adaptive approach to manage class separability, rather than a one-size-fits-all method, is crucial for improving accuracy.
- Efficiency can go hand-in-hand with performance: The paper demonstrates that it's possible to achieve state-of-the-art results without resorting to computationally intensive methods like prompt tuning, which often require extensive pre-training and large memory footprints.
Key Results
AMEND achieved state-of-the-art performance on five out of seven datasets tested, including the challenging Herbarium-19 dataset. For the remaining two datasets, it performed comparably to the state-of-the-art. Specifically:
- On the highly imbalanced Herbarium-19 dataset, AMEND achieved 44.2% accuracy, significantly outperforming the next best method (PromptCAL at 37%).
- For the Stanford Cars dataset, AMEND improved performance by 3.2% over its baseline with neighbors, achieving state-of-the-art results.
- The framework is also significantly more memory and hardware efficient than comparable methods like PromptCAL, requiring less GPU memory and no prompt warmup training.
Limitations and Future Work
- Hyperparameter Tuning: While the performance of AMEND is shown to be stable with varying numbers of neighbors, finding the optimal number of neighbors is suggested as a future research direction.
- Dataset Splits: The paper assumes that the total number of classes in the unlabeled data is known or can be estimated. Future work could explore methods that do not require this prior knowledge.
Summary updated