PMCE: Probabilistic Multi-Granularity Semantics with Caption-Guided Enhancement for Few-Shot Learning

Jiaying Wu, Can Gao, Jinglu Hu, Hui Li, Xiaofeng Cao, Jingcai GuoarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces PMCE, a novel framework for few-shot learning that tackles the problem of biased prototypes and weak query representations by leveraging multi-granularity semantics. By combining class-level semantic priors with instance-level caption guidance, PMCE significantly improves recognition accuracy, especially in challenging 1-shot scenarios.

What Problem Does This Paper Address?

Few-shot learning (FSL) aims to recognize new object categories using only a few labeled examples. A core challenge in FSL is that prototypes (representations of a class) derived from scarce data are often unreliable and lead to poor generalization. While semantic information (like class names) has been used to improve FSL, existing methods often apply it only to support set prototypes, leaving query images (the ones to be classified) with less refined representations. This can lead to a mismatch, especially when dealing with visually similar classes or limited data (1-shot learning), where even small biases can have a large impact. PMCE addresses this by proposing a more comprehensive use of semantics, both at the class level for prior calibration and at the instance level for feature enhancement of both support and query images.

Methods

PMCE introduces a two-pronged semantic approach:

  1. Semantics-Guided Prior Selection for MAP Calibration:
    • A knowledge bank is built from base classes, storing their visual statistics (mean features) and CLIP-encoded class name embeddings.
  2. At meta-test time, for a novel class, its CLIP-encoded name embedding is used to find the most semantically similar base classes.
  3. The visual means of these similar base classes are aggregated into a class-specific prior.
  4. This prior is then fused with the initial support set prototype using a Maximum A Posteriori (MAP) update, effectively "calibrating" the prototype with semantic knowledge. This is a probabilistic approach that balances the evidence from the few support examples with the semantic prior.
  5. Caption-Guided Dual-Stream Enhancement:
    • A frozen BLIP captioner generates label-free, instance-level descriptions for both support and query images.
  6. A lightweight "enhancer" module (a cross-attention mechanism) uses these captions as semantic context to refine both the support prototypes and the query features. This ensures that both components of the classification process are enhanced by instance-specific semantic information.
  7. A consistency regularizer is used to mitigate noise in the generated captions.

Crucially, PMCE keeps the visual backbone and vision-language encoders frozen, training only the enhancer module. This means improvements come from how semantics are injected, not from increasing model complexity.

Datasets

The paper evaluates PMCE on four standard few-shot learning benchmarks:

  • MiniImageNet: A subset of ImageNet with 100 classes.
  • TieredImageNet: Another subset of ImageNet, with a hierarchical class structure.
  • CIFAR-FS: A split of CIFAR-100.
  • FC100: A split of CIFAR-100 with a superclass-based structure to reduce semantic overlap.

Key Insights and Ideas

  • Multi-granularity semantics are key: Using both coarse class-level semantics (for prior selection) and fine-grained instance-level semantics (from captions for feature enhancement) is more effective than relying on one alone.
  • Captioning for everyone: Applying label-free captions to both support and query images, and using them to enhance both prototypes and query features, is crucial for aligning representations in the same metric space.
  • Semantics guide priors, visuals provide evidence: The semantic information (class names) acts as a guide to select relevant base-class priors, which are then probabilistically fused with the visual evidence from the few support examples. This is a more principled way to leverage semantic knowledge than directly embedding it into features.
  • Frozen components are sufficient: Significant gains can be achieved by intelligently injecting semantic information into existing architectures without needing to fine-tune large pre-trained models or significantly increase model capacity.

Key Results

  • PMCE consistently outperforms strong semantic and vision-language baselines across all four benchmarks.
  • The most significant gains are observed in the 1-shot setting, where prototype bias is most problematic.
  • On MiniImageNet (1-shot), PMCE achieves an absolute gain of 7.71% over the strongest semantic competitor (SemFew-Trans) when using the Swin-T backbone, reaching 85.03%.
  • Qualitative results (Grad-CAM visualizations) show that caption guidance leads to more focused attention on object regions and suppresses background noise in query images.
  • Prototype evolution visualizations (t-SNE) demonstrate that PMCE effectively shifts prototypes towards ground-truth centers and makes clusters more compact.

Limitations and Future Work

  • Caption quality: The paper acknowledges that caption quality can vary, and suggests future work could explore using caption confidence to further reduce semantic drift under noisy semantics.
  • Cross-domain performance: While PMCE shows comparable performance to strong baselines in cross-domain settings, the absolute accuracies are lower, indicating that domain gaps still pose a challenge. The authors suggest that semantic cross-attention offers a modest but consistent benefit.
  • Computational cost: While the enhancer is lightweight, generating captions adds an inference step. Future work could explore more efficient ways to leverage instance-level semantics.

Summary updated

Abstract

Few-shot learning aims to identify novel categories from only a handful of labeled samples, where prototypes estimated from scarce data are often biased and generalize poorly. Semantic-based methods alleviate this by introducing coarse class-level information, but they are mostly applied on the support side, leaving query representations unchanged. In this paper, we present PMCE, a Probabilistic few-shot framework that leverages Multi-granularity semantics with Caption-guided Enhancement. PMCE constructs a nonparametric knowledge bank that stores visual statistics for each category as well as CLIP-encoded class name embeddings of the base classes. At meta-test time, the most relevant base classes are retrieved based on the similarities of class name embeddings for each novel category. These statistics are then aggregated into category-specific prior information and fused with the support set prototypes via a simple MAP update. Simultaneously, a frozen BLIP captioner provides label-free instance-level image descriptions, and a lightweight enhancer trained on base classes optimizes both support prototypes and query features under an inductive protocol with a consistency regularization to stabilize noisy captions. Experiments on four benchmarks show that PMCE consistently improves over strong baselines, achieving up to 7.71% absolute gain over the strongest semantic competitor on MiniImageNet in the 1-shot setting. Our code is available at https://anonymous.4open.science/r/PMCE-275D
BibTeX
@misc{wu2026pmce,
  author = {Jiaying Wu and Can Gao and Jinglu Hu and Hui Li and Xiaofeng Cao and Jingcai Guo},
  title = {{PMCE: Probabilistic Multi-Granularity Semantics with Caption-Guided Enhancement for Few-Shot Learning}},
  year = {2026},
  eprint = {2601.14111},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.14111},
}
Figure