Abstract
Few-shot classification requires adapting knowledge learned from a large annotated base dataset to recognize novel unseen classes, each represented by few labeled examples. In such a scenario, pretraining a network with high capacity on the large dataset and then finetuning it on the few examples causes severe overfitting. At the same time, training a simple linear classifier on top of "frozen" features learned from the large labeled dataset fails to adapt the model to the properties of the novel classes, effectively inducing underfitting. In this paper we propose an alternative approach to both of these two popular strategies. First, our method pseudo-labels the entire large dataset using the linear classifier trained on the novel classes. This effectively "hallucinates" the novel classes in the large dataset, despite the novel categories not being present in the base database. Then, it finetunes the entire model with a distillation loss on the pseudo-labeled base examples, in addition to the standard cross-entropy loss on the novel dataset. This step effectively trains the network to recognize contextual and appearance cues that are useful for the novel-category recognition but using the entire large-scale base dataset and thus overcoming the inherent data-scarcity problem of few-shot learning. Despite the simplicity of the approach, we show that our method outperforms the state-of-the-art on four well-established few-shot classification benchmarks.
Introduction
Deep learning models are notoriously data-hungry, typically requiring thousands of examples per class to generalize well. In Few-Shot Learning (FSL), the goal is to teach a model to recognize new categories (novel classes) using only 1 to 5 examples. Traditionally, researchers face a "Goldilocks" problem of model capacity:
- Overfitting: Finetuning a high-capacity network on a handful of images leads the model to memorize specific pixels rather than learning generalizable features.
- Underfitting: Freezing the pretrained backbone and training only a linear classifier prevents the model from adapting its internal representations to the specific nuances of the new classes.
This paper introduces Label Hallucination, a strategy that allows for safe, full-model finetuning by leveraging the large, disjoint base dataset as a proxy for the novel classes.
Methodology
The Label Hallucination framework consists of three primary stages following standard pretraining on a base dataset $C^{base}$.
1. Initial Transfer
First, a model is pretrained on the large base dataset. During a specific few-shot task (iteration $i$), the backbone $\Theta^{base}$ is frozen, and a simple linear classifier $\phi_i$ is trained on the few available labeled examples from the novel support set $C^{novel}$.
2. The Hallucination Step
The trained linear classifier $\phi_i$ is then applied to the entire original base dataset. Although the base dataset does not contain the novel classes, the classifier assigns "soft" pseudo-labels (probability distributions) to every base image. For instance, a base image of a "tiger" might be pseudo-labeled with a high probability for a novel "domestic cat" class due to visual similarity. This process "hallucinates" the presence of novel classes within the large-scale data.
3. Full-Model Finetuning
Finally, the entire network (both the backbone $\Theta'$ and the classifier $\phi'_i$) is finetuned. The training uses a dual loss function: a standard cross-entropy loss for the few novel images and a distillation loss that encourages the model to match the "hallucinated" soft pseudo-labels on the large base dataset. This effectively treats the massive base dataset as a form of informed data augmentation, providing the variety needed to update network weights without catastrophic overfitting.
Experimental Results
The method was evaluated on four standard benchmarks: miniImageNet, tieredImageNet, CIFAR-FS, and FC100. Label Hallucination achieved state-of-the-art (SOTA) results across all settings.
As shown in the results table, the method reached 68.28% accuracy in 1-shot and 86.54% in 5-shot settings on miniImageNet, outperforming previous leading methods like RFS and SKD. On the larger tieredImageNet, it set new benchmarks of 73.34% (1-shot) and 87.68% (5-shot). The performance gains were consistent regardless of the specific pretraining strategy employed.
Key Insights and Analysis
The Importance of Soft Labels
A critical finding of the study is that "soft" labels are essential. In ablation studies, "hard" pseudo-labels (assigning a single definitive class) performed significantly worse. Soft labels allow the model to capture nuanced similarities between base objects and novel categories, which is vital since the classes are disjoint.
Backbone Adaptation
Ablation studies confirmed that the majority of performance gains stem from updating the backbone (the feature extractor) rather than just the classifier head. By using the hallucinated labels on the base dataset, the model successfully adapts its internal features to be more discriminative for the novel task.
Leveraging Disjoint Data
The research demonstrates that a dataset containing completely different classes can still be highly beneficial for finetuning. Contextual cues in the base set (e.g., a "street scene") provide useful information for novel classes (e.g., "pedestrian"), even if the original labels were unrelated.
Conclusion and Limitations
Label Hallucination provides a simple yet effective way to bridge the gap between underfitting and overfitting in few-shot classification. By repurposing large-scale base data through pseudo-labeling, it enables the adaptation of high-capacity models to data-scarce tasks.
However, the method does have limitations. Because the model must be finetuned for every new task (episode), the computational cost at inference time is higher than methods using a frozen backbone. Additionally, the effectiveness of the method may depend on the domain gap between the base and novel datasets; while it works well across natural image datasets, its performance on cross-domain tasks (e.g., natural images to medical imagery) remains an area for future work.
Summary updated