Envisioning Class Entity Reasoning by Large Language Models for Few-shot Learning

Mushui Liu, Fangtai Wu, Bozheng Li, Ziqian Lu, Yunlong Yu, Xi Li

TL;DR

Researchers from Zhejiang University have developed ECER-FSL, a framework that uses Large Language Models (LLMs) to break down abstract category names (like "Newfoundland dog") into concrete visual entities (like "thick coat" or "black fur"). By "envisioning" these specific traits, the model can recognize new objects from as little as a single image, outperforming current state-of-the-art methods by nearly 2%.

What Problem Does This Paper Address?

Few-Shot Learning (FSL) is the "holy grail" of computer vision: training a model to recognize a new category using only one or five examples. While humans can do this easily by combining visual cues with prior knowledge, machines struggle because a single image rarely captures all the defining characteristics of a class.

Existing methods often try to use "semantic information"—usually just the class name—to help the model. However, a name like "Newfoundland" is too abstract. It doesn't tell the model what to look for in the pixels. Furthermore, when models try to generate descriptions, they often "hallucinate" irrelevant details, and there is a significant "misalignment" between high-level text descriptions and low-level visual features.

Methods

The authors introduce a two-stage approach called ECER-FSL (Entities-Assisted Class Entity Reasoning for Few-Shot Learning):

  1. Multi-Modality Pre-training: Before looking at new classes, the visual backbone is trained using image-caption pairs generated by an off-the-shelf model (CogVLM). This teaches the encoder to align local image regions with specific text descriptions from the start.
  2. Entity Generation & Filtering: For a new class, the system asks an LLM (GPT-4o) to list $N$ relevant visual entities. To prevent "hallucinations," they use a Selection Strategy that filters out entities with low textual similarity to the original class name.
  3. Progressive Visual-Semantic Aggregation (PVSA): This is the core engine, consisting of two modules:
  4. SVPE (Semantic-guided Visual Pattern Extraction): This module "scans" the image features at multiple stages of the neural network, looking specifically for the patterns described by the LLM-generated entities.
  5. PC (Prototype-Calibration): This module takes those extracted visual patterns and uses them to "calibrate" or refine the class prototype (the mathematical representation of the category), making it much more robust than a prototype based on a single image alone.

Datasets

The model was evaluated on two fronts:

  • Traditional FSL: miniImageNet, tieredImageNet, CIFAR-FS, and FC100.
  • Cross-Domain FSL (BSCD-FSL): This tests the model's ability to generalize to completely different types of images, including ChestX (X-rays), ISIC (skin lesions), EuroSAT (satellite imagery), and CropDisease (plant leaves).

Key Insights and Ideas

  • LLMs as a "Mental Bridge": The key "aha" moment is that LLMs possess vast world knowledge that can fill in the gaps of a single image. If a photo of a dog is taken from the front, the LLM can remind the model to also consider "thick fur" or "massive build," which might be present but not obvious.
  • Granularity Matters: Moving from class-level semantics (the name) to entity-level semantics (the traits) provides the "fine-grained" guidance necessary for the model to distinguish between very similar classes.
  • Multi-Stage Infusion: The researchers found that injecting semantic guidance at multiple blocks of the vision backbone (Stages 1 through 4) consistently improved performance, suggesting that "reasoning" about entities happens at both low-level (texture) and high-level (shape) visual scales.

Key Results

  • State-of-the-Art Performance: On the 5-way 1-shot task, ECER-FSL achieved an average improvement of 1.95% over the previous best competitor across four major benchmarks.
  • Robustness in New Domains: In cross-domain tasks, the model showed even more significant gains, outperforming competitors by up to 3.71% on the ISIC (medical) dataset.
  • Efficiency: Ablation studies showed that using just 10 entities from the LLM is the "sweet spot"—adding more doesn't necessarily help, as the most critical visual traits are usually captured in the top 10.

Limitations and Future Work

While the filtering strategy helps, the model is still somewhat dependent on the quality of the LLM's output. If an LLM provides traits that are visually ambiguous or completely absent from the support set, the "calibration" might be less effective.

The authors suggest that future work could involve even tighter alignment between the reasoning capabilities of LLMs and the feature extraction process of vision models, perhaps moving toward a more unified "reasoning-while-seeing" architecture.

Summary updated

Abstract

Few-shot learning (FSL) aims to recognize new concepts using a limited number of visual samples. Existing approaches attempt to incorporate semantic information into the limited visual data for category understanding. However, these methods often enrich class-level feature representations with abstract category names, failing to capture the nuanced features essential for effective generalization. To address this issue, we propose a novel framework for FSL, which incorporates both the abstract class semantics and the concrete class entities extracted from Large Language Models (LLMs), to enhance the representation of the class prototypes. Specifically, our framework composes a Semantic-guided Visual Pattern Extraction (SVPE) module and a Prototype-Calibration (PC) module, where the SVPE meticulously extracts semantic-aware visual patterns across diverse scales, while the PC module seamlessly integrates these patterns to refine the visual prototype, enhancing its representativeness. Extensive experiments on four few-shot classification benchmarks and the BSCD-FSL cross-domain benchmarks showcase remarkable advancements over the current state-of-the-art methods. Notably, for the challenging one-shot setting, our approach, utilizing the ResNet-12 backbone, achieves an impressive average improvement of 1.95% over the second-best competitor.
BibTeX
@inproceedings{liu2025envisioning,
  author = {Mushui Liu and Fangtai Wu and Bozheng Li and Ziqian Lu and Yunlong Yu and Xi Li},
  title = {{Envisioning Class Entity Reasoning by Large Language Models for Few-shot Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i18.34081},
}

Loading related content...

Figure