Abstract
Few-shot learning (FSL) aims to recognize new categories given limited training samples. The core challenge is to avoid overfitting to the minimal data while ensuring good generalization to novel classes. One mainstream method employs prototypes from visual feature extractors as classifier weights, and the performance depends on the quality of the prototype. Since different categories may have similar visual features, the visual prototype has limitations. This is because existing methods only learn a simple visual feature extractor during the pre-training stage but neglect the importance of a well-developed feature space for the prototype. We introduce the Semantic Enhanced Visual Prototype framework (SEVpro) to address this issue. SEVpro refines prototype learning from the pre-training stage and serves as a versatile plug-and-play framework for all prototype-based FSL methods. Specifically, we enhance prototype discriminability by transforming semantic embeddings into the visual space, aiding in separating categories with similar visual features. For novel class learning, we leverage knowledge from base classes and incorporate semantic information to elevate prototype quality further. Meanwhile, extensive experiments on FSL benchmarks and ablation studies demonstrate the superiority of our proposed SEVpro for FSL.
Introduction
Few-Shot Learning (FSL) is a critical frontier in artificial intelligence, aiming to bridge the gap between human-like learning—where a single example can suffice for recognition—and traditional deep learning, which requires thousands of labeled samples. The standard paradigm for FSL involves two distinct phases: pre-training a robust feature extractor on a large "base" dataset and then adapting a classifier to "novel" classes using only a few examples (the "support set").
A primary bottleneck in this approach is visual ambiguity. In a purely visual feature space, objects from different categories may appear nearly identical. For instance, an egg and a ping pong ball share similar shapes, colors, and textures. Without additional context, a visual feature extractor might map them to overlapping regions in the embedding space, leading to "noisy" prototypes and classification errors. SEVpro addresses this by leveraging the inherent discriminative power of language, treating semantic descriptions as a "spacer" to better organize the visual feature space.
Methodology
SEVpro is designed as a "plug-and-play" framework that can be integrated into existing prototype-based FSL models. It operates across both the pre-training and inference stages.
1. Pre-training with Semantic Mapping
During the initial training phase, SEVpro aligns visual features with semantic embeddings. Class labels are converted into descriptive sentences using templates (e.g., "A photo of an [egg]") and processed through a pre-trained CLIP text encoder. These high-dimensional text vectors are then mapped into the visual feature space via a linear transformation.
To optimize this alignment, the model employs a Hybrid Contrastive Loss. In addition to standard Cross-Entropy Loss for classification, it uses an infoNCE loss. This contrastive mechanism "pulls" an image's visual features toward its corresponding semantic vector while "pushing" it away from the vectors of other categories. This ensures that even visually similar objects are separated in the feature space based on their distinct linguistic identities.
2. Few-Shot Inference and Semantic Guidance
In the few-shot phase, SEVpro enhances the quality of prototypes for novel classes through two mechanisms:
- Semantic Attention: The model generates a mask using a Hadamard product to emphasize visual features that are most relevant to the category's linguistic meaning.
- Knowledge Transfer: It utilizes an attention mechanism to "borrow" knowledge from visually similar base classes. For example, if the model is learning to identify a "zebra" for the first time, it can leverage its existing knowledge of "horses" to refine the zebra prototype.
Experiments and Results
The framework was rigorously evaluated on three standard FSL benchmarks: miniImageNet, tieredImageNet, and CIFAR-FS. SEVpro achieved state-of-the-art (SOTA) performance across all datasets, with particularly notable gains in the challenging 1-shot scenario:
- miniImageNet (1-shot): Achieved 71.81% accuracy, outperforming the baseline SEGA model by nearly 3%.
- tieredImageNet (1-shot): Reached 72.77% accuracy.
- CIFAR-FS (1-shot): Reached 80.36% accuracy.
t-SNE visualizations confirmed that SEVpro produces tighter, more distinct clusters compared to traditional methods. By forcing the visual extractor to align with semantic descriptions during pre-training, the resulting feature space becomes more structured and resilient to visual noise.
Conclusion
SEVpro demonstrates that the foundation of few-shot learning lies in the quality of the initial feature space. By integrating language as a guiding signal during pre-training, the framework effectively resolves visual ambiguities that plague purely vision-based models. While the current implementation uses simple text templates, future work may explore richer, more descriptive semantic inputs to further enhance prototype discriminability. Despite the added complexity of incorporating a text encoder during training, the "plug-and-play" nature of SEVpro makes it a versatile and powerful tool for advancing the state of few-shot recognition.
Summary updated