Abstract
Few-shot learning (FSL) aims to recognize novel categories from limited examples. While traditional methods rely on visual prototypes, visual ambiguity often leads to classification errors. We present SEVpro (Semantic Enhanced Visual Prototype), a framework that leverages language as a spatial anchor during pre-training. By aligning visual features with semantic embeddings from CLIP, SEVpro creates a more discriminative feature space. Our approach achieves state-of-the-art results on benchmarks like miniImageNet and CIFAR-FS, demonstrating the power of early-stage semantic integration.
Introduction: The Visual Ambiguity Problem
Humans can distinguish between an egg and a ping pong ball with ease, even from a single glance. However, for an AI model relying solely on low-dimensional visual features, these objects may appear nearly identical. This "cluttered" embedding space is a major bottleneck in few-shot learning (FSL).
Most existing FSL methods attempt to fix this during the fine-tuning stage. SEVpro argues that the real solution lies earlier, in the pre-training stage. If the initial feature extractor isn't taught to separate these categories early on, no amount of late-stage fine-tuning can fully fix a cluttered embedding space.
The SEVpro Framework
SEVpro is a plug-and-play framework designed to refine how models learn visual features. It operates in two main stages:
1. Semantically Enhanced Pre-training
Instead of training a standard image classifier, SEVpro uses a contrastive loss (infoNCE). Class labels are converted into descriptive sentences (e.g., "A photo of a [label]") and encoded using a frozen CLIP text encoder. During training, the model is forced to pull visual features closer to their corresponding text embeddings while pushing them away from others. This "stretches" the visual space using semantic knowledge as a guide.
2. Semantic-Guided Classifier Learning
When faced with a new "novel" class, the model doesn't just average the few available images. It uses an attention mechanism to transfer relevant knowledge from known "base" classes (e.g., using "horse" knowledge to help identify a "zebra"). It then applies a semantic guidance mask (via a Hadamard product) to the visual prototype, emphasizing the most important features based on the class's word embedding.
Key Results and Insights
SEVpro achieved state-of-the-art (SOTA) or near-SOTA performance across all benchmarks, particularly in the most difficult 1-shot scenarios:
- miniImageNet (5-way 1-shot): Reached 71.81% accuracy, outperforming the strong SEGA baseline (69.04%).
- CIFAR-FS (5-way 1-shot): Achieved 80.36%, a significant lead over previous methods.
- Ablation Studies: The authors found that the infoNCE contrastive loss was the most effective at organizing the embedding space compared to triplet loss or cosine similarity.
The most important insight is that language provides a structured prior that vision lacks. By forcing visual features to align with a pre-existing semantic space, the model avoids "collapsing" different categories into the same visual cluster. SEVpro's versatility was further proven by adding it to existing models like ProtoNet, which saw significant performance jumps.
Conclusion
SEVpro demonstrates that injecting semantic intelligence into the initial feature extraction phase yields much higher returns than late-stage fine-tuning. While the current model uses simple templates, future work could explore richer descriptions to provide even better guidance. SEVpro's success highlights the potential of multi-modal alignment as a foundation for more robust and human-like AI recognition.
Summary updated