TL;DR
This paper introduces PMCE, a novel framework for few-shot learning that tackles the problem of biased prototypes and weak query representations by leveraging multi-granularity semantics. By combining class-level semantic priors with instance-level caption guidance, PMCE significantly improves recognition accuracy, especially in challenging 1-shot scenarios.
What Problem Does This Paper Address?
Few-shot learning (FSL) aims to recognize new object categories using only a few labeled examples. A core challenge in FSL is that prototypes (representations of a class) derived from scarce data are often unreliable and lead to poor generalization. While semantic information (like class names) has been used to improve FSL, existing methods often apply it only to support set prototypes, leaving query images (the ones to be classified) with less refined representations. This can lead to a mismatch, especially when dealing with visually similar classes or limited data (1-shot learning), where even small biases can have a large impact. PMCE addresses this by proposing a more comprehensive use of semantics, both at the class level for prior calibration and at the instance level for feature enhancement of both support and query images.
Methods
PMCE introduces a two-pronged semantic approach:
- Semantics-Guided Prior Selection for MAP Calibration:
- A knowledge bank is built from base classes, storing their visual statistics (mean features) and CLIP-encoded class name embeddings.
- At meta-test time, for a novel class, its CLIP-encoded name embedding is used to find the most semantically similar base classes.
- The visual means of these similar base classes are aggregated into a class-specific prior.
- This prior is then fused with the initial support set prototype using a Maximum A Posteriori (MAP) update, effectively "calibrating" the prototype with semantic knowledge. This is a probabilistic approach that balances the evidence from the few support examples with the semantic prior.
- Caption-Guided Dual-Stream Enhancement:
- A frozen BLIP captioner generates label-free, instance-level descriptions for both support and query images.
- A lightweight "enhancer" module (a cross-attention mechanism) uses these captions as semantic context to refine both the support prototypes and the query features. This ensures that both components of the classification process are enhanced by instance-specific semantic information.
- A consistency regularizer is used to mitigate noise in the generated captions.
Crucially, PMCE keeps the visual backbone and vision-language encoders frozen, training only the enhancer module. This means improvements come from how semantics are injected, not from increasing model complexity.
Datasets
The paper evaluates PMCE on four standard few-shot learning benchmarks:
- MiniImageNet: A subset of ImageNet with 100 classes.
- TieredImageNet: Another subset of ImageNet, with a hierarchical class structure.
- CIFAR-FS: A split of CIFAR-100.
- FC100: A split of CIFAR-100 with a superclass-based structure to reduce semantic overlap.
Key Insights and Ideas
- Multi-granularity semantics are key: Using both coarse class-level semantics (for prior selection) and fine-grained instance-level semantics (from captions for feature enhancement) is more effective than relying on one alone.
- Captioning for everyone: Applying label-free captions to both support and query images, and using them to enhance both prototypes and query features, is crucial for aligning representations in the same metric space.
- Semantics guide priors, visuals provide evidence: The semantic information (class names) acts as a guide to select relevant base-class priors, which are then probabilistically fused with the visual evidence from the few support examples. This is a more principled way to leverage semantic knowledge than directly embedding it into features.
- Frozen components are sufficient: Significant gains can be achieved by intelligently injecting semantic information into existing architectures without needing to fine-tune large pre-trained models or significantly increase model capacity.
Key Results
- PMCE consistently outperforms strong semantic and vision-language baselines across all four benchmarks.
- The most significant gains are observed in the 1-shot setting, where prototype bias is most problematic.
- On MiniImageNet (1-shot), PMCE achieves an absolute gain of 7.71% over the strongest semantic competitor (SemFew-Trans) when using the Swin-T backbone, reaching 85.03%.
- Qualitative results (Grad-CAM visualizations) show that caption guidance leads to more focused attention on object regions and suppresses background noise in query images.
- Prototype evolution visualizations (t-SNE) demonstrate that PMCE effectively shifts prototypes towards ground-truth centers and makes clusters more compact.
Limitations and Future Work
- Caption quality: The paper acknowledges that caption quality can vary, and suggests future work could explore using caption confidence to further reduce semantic drift under noisy semantics.
- Cross-domain performance: While PMCE shows comparable performance to strong baselines in cross-domain settings, the absolute accuracies are lower, indicating that domain gaps still pose a challenge. The authors suggest that semantic cross-attention offers a modest but consistent benefit.
- Computational cost: While the enhancer is lightweight, generating captions adds an inference step. Future work could explore more efficient ways to leverage instance-level semantics.
Summary updated