Envisioning Class Entity Reasoning by Large Language Models for Few-shot Learning

Mushui Liu, Fangtai Wu, Bozheng Li, Ziqian Lu, Yunlong Yu, Xi Li

Abstract

Few-shot learning (FSL) aims to recognize new concepts using a limited number of visual samples. Existing approaches attempt to incorporate semantic information into the limited visual data for category understanding. However, these methods often enrich class-level feature representations with abstract category names, failing to capture the nuanced features essential for effective generalization. To address this issue, we propose a novel framework for FSL, which incorporates both the abstract class semantics and the concrete class entities extracted from Large Language Models (LLMs), to enhance the representation of the class prototypes. Specifically, our framework composes a Semantic-guided Visual Pattern Extraction (SVPE) module and a Prototype-Calibration (PC) module, where the SVPE meticulously extracts semantic-aware visual patterns across diverse scales, while the PC module seamlessly integrates these patterns to refine the visual prototype, enhancing its representativeness. Extensive experiments on four few-shot classification benchmarks and the BSCD-FSL cross-domain benchmarks showcase remarkable advancements over the current state-of-the-art methods. Notably, for the challenging one-shot setting, our approach, utilizing the ResNet-12 backbone, achieves an impressive average improvement of 1.95% over the second-best competitor.

Introduction

Few-Shot Learning (FSL) represents a significant challenge in computer vision: enabling models to recognize new categories from just one or five examples. While humans leverage prior knowledge to identify defining traits, machines often struggle because a single image rarely captures the full scope of a category's characteristics.

Existing FSL methods typically utilize class names as semantic guidance. However, abstract names like "Newfoundland" are often insufficient for guiding pixel-level feature extraction. There is a notable "misalignment" between high-level text and low-level visual features. To bridge this gap, the proposed framework uses Large Language Models (LLMs) to decompose abstract categories into concrete visual entities (e.g., "thick coat," "black fur"), providing the model with a "mental bridge" to envision missing traits.

Conceptual overview of class entity reasoning using LLMs to decompose abstract categories into concrete visual patterns.

Methodology: ECER-FSL

The ECER-FSL (Entities-Assisted Class Entity Reasoning for Few-Shot Learning) framework operates through three primary stages:

1. Multi-Modality Pre-training

The visual backbone is initially trained using image-caption pairs generated by an off-the-shelf model (CogVLM). This step ensures the encoder can align local image regions with specific textual descriptions, establishing a foundation for visual-semantic reasoning.

2. Entity Generation and Filtering

For a target class, the system prompts an LLM to generate $N$ relevant visual entities. To mitigate hallucinations, a selection strategy filters these entities based on their textual similarity to the original class name, ensuring only the most relevant traits are used for reasoning.

3. Progressive Visual-Semantic Aggregation (PVSA)

The core of the framework is the PVSA engine, which integrates semantic guidance across multiple stages of the vision backbone. As shown in the architecture below, it consists of two key modules:

  • Semantic-guided Visual Pattern Extraction (SVPE): This module scans image features at various network blocks to identify patterns corresponding to the LLM-generated entities.
  • Prototype-Calibration (PC): This module aggregates the extracted patterns to refine the visual prototype, making it more robust and representative of the class than a prototype derived from a single image.
The ECER-FSL architecture, featuring the multi-stage SVPE blocks and the PC module for prototype refinement.

Experimental Results

The model was evaluated on standard FSL benchmarks (miniImageNet, tieredImageNet, CIFAR-FS, FC100) and the BSCD-FSL cross-domain benchmarks (ChestX, ISIC, EuroSAT, CropDisease).

  • State-of-the-Art Performance: In the 5-way 1-shot setting, ECER-FSL achieved an average improvement of 1.95% over the previous best competitors across four major benchmarks.
  • Cross-Domain Robustness: The model demonstrated significant gains in specialized domains, outperforming others by up to 3.71% on the ISIC medical dataset, proving the value of LLM-based reasoning in diverse contexts.
  • Efficiency: Ablation studies indicate that using approximately 10 entities is the "sweet spot"; adding more does not necessarily improve performance as the most critical visual traits are typically captured within the top 10.

Conclusion

By shifting from abstract class-level semantics to granular entity-level reasoning, ECER-FSL provides vision models with the necessary guidance to excel in data-scarce environments. Future work will explore deeper integration of LLM reasoning directly into the feature extraction process, moving toward unified "reasoning-while-seeing" architectures.

Summary updated

Abstract

Few-shot learning (FSL) aims to recognize new concepts using a limited number of visual samples. Existing approaches attempt to incorporate semantic information into the limited visual data for category understanding. However, these methods often enrich class-level feature representations with abstract category names, failing to capture the nuanced features essential for effective generalization. To address this issue, we propose a novel framework for FSL, which incorporates both the abstract class semantics and the concrete class entities extracted from Large Language Models (LLMs), to enhance the representation of the class prototypes. Specifically, our framework composes a Semantic-guided Visual Pattern Extraction (SVPE) module and a Prototype-Calibration (PC) module, where the SVPE meticulously extracts semantic-aware visual patterns across diverse scales, while the PC module seamlessly integrates these patterns to refine the visual prototype, enhancing its representativeness. Extensive experiments on four few-shot classification benchmarks and the BSCD-FSL cross-domain benchmarks showcase remarkable advancements over the current state-of-the-art methods. Notably, for the challenging one-shot setting, our approach, utilizing the ResNet-12 backbone, achieves an impressive average improvement of 1.95% over the second-best competitor.
BibTeX
@inproceedings{liu2025envisioning,
  author = {Mushui Liu and Fangtai Wu and Bozheng Li and Ziqian Lu and Yunlong Yu and Xi Li},
  title = {{Envisioning Class Entity Reasoning by Large Language Models for Few-shot Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i18.34081},
}
Figure