Less is more: A closer look at semantic-based few-shot learning

Chunpeng Zhou, Zhi Yu, Xilu Yuan, Sheng Zhou, Jiajun Bu, Haishuai Wang

TL;DR

This paper proposes a simple yet effective framework for few-shot learning (FSL) that leverages the power of pre-trained language models (LMs) by directly adding visual and textual features. Their approach achieves state-of-the-art results, particularly in the challenging 1-shot learning scenario, by emphasizing the generalization capabilities of LMs and using learnable prompts.

What Problem Does This Paper Address?

Few-shot learning (FSL) aims to enable models to recognize new categories with very limited training examples. This is a significant challenge because traditional deep learning models struggle with insufficient supervision. While recent research has explored using semantic information from LMs to aid FSL, many approaches rely on complex fusion mechanisms that can inadvertently hinder the LM's inherent generalization power. This paper argues that the full potential of pre-trained LMs in FSL has been underestimated and proposes a simpler method to harness this capability.

Methods

The core of the proposed framework, SimpleFSL, is remarkably straightforward. It consists of three main components:

  • Visual Backbone: Extracts features from input images.
  • Textual Backbone (Pre-trained LM): Extracts semantic features from textual descriptions of classes.
  • Multi-modal Feature Fusion: Instead of complex fusion modules, SimpleFSL directly adds the visual and textual features. This simple "add operation" is chosen to minimize interference with the LM's generalization capabilities.

To further enhance performance, SimpleFSL++ introduces:

  • Learnable Prompts: Instead of fixed prompts (e.g., "A photo of a [class name]"), they use learnable prompt vectors that are optimized during meta-training. This allows for better adaptation to different datasets.
  • Self-Ensemble and Self-Distillation: These techniques are employed to further refine the model's predictions and improve robustness.

The authors also explore different adaptor mechanisms to align the textual and visual feature spaces and investigate various prompt designs (dataset-aware, class-aware, task-aware), ultimately finding dataset-aware prompts to be most effective.

Datasets

The paper evaluates its methods on four widely used FSL benchmarks:

  • miniImageNet: A subset of ImageNet, commonly used for FSL research.
  • tieredImageNet: A larger dataset with more semantically distinct base and novel classes.
  • CIFAR-FS: A dataset derived from CIFAR-100, known for its limited number of images per class.
  • FC100: Another dataset derived from CIFAR-100, presenting a steeper challenge due to its semantic superclasses.

Key Insights and Ideas

  • Simplicity is Key: The paper champions the idea that complex fusion mechanisms are not always necessary and can even be detrimental in semantic-based FSL. A simple addition of features can be highly effective.
  • LM Generalization Power: The core argument is that pre-trained LMs possess significant generalization capabilities that are often overlooked. Their method directly leverages this power.
  • Learnable Prompts Matter: Moving from fixed prompts to learnable ones significantly improves adaptability and performance across different datasets.
  • Dataset-Aware Prompts are Optimal: While class-aware and task-aware prompts were explored, the dataset-aware approach proved most effective, suggesting that a balance between specificity and generality is crucial.

Key Results

  • State-of-the-Art Performance: SimpleFSL and SimpleFSL++ achieve impressive results across all four datasets, consistently outperforming existing methods.
  • 1-Shot Learning Dominance: The proposed framework shows particularly strong performance in the 1-shot learning task, surpassing the state-of-the-art by an average of 3.3% in classification accuracy on miniImageNet.
  • Ablation Studies Confirm Efficacy: Experiments demonstrate that each component of their framework (visual backbone, LM, learnable prompts, self-ensemble, self-distillation) contributes to the overall performance improvement.
  • Visualization Support: t-SNE visualizations show that SimpleFSL++ achieves more distinct class separability compared to a sole visual backbone or more complex methods like SP-CLIP.

Limitations and Future Work

  • Prompt Design Exploration: While dataset-aware prompts were found to be optimal, the authors suggest that further research into designing more effective prompts for FSL is a promising avenue.
  • Adaptor Choice: The paper explores different adaptor mechanisms, noting that their MLP with a bottleneck structure shows a slight advantage due to fewer parameters, but further investigation into adaptors could be beneficial.
  • Fusion Mechanism Simplicity: While the simple "add" operation is effective, the paper acknowledges that other fusion mechanisms (like concatenation or attention) were explored and showed negligible differences, suggesting that simplicity is a strong choice.

Summary updated

Abstract

Highlights•Previous Semantic-based few-shot methods focus on designing complex fusion modules, while ignoring the generalization capacity of language models.•We propose a simple framework, which fully exploits the LM with learnable prompts.•Ours performance in 1-shot learning surpasses the current SOTA by 3.3% in accuracy.
BibTeX
@article{zhou2025less,
  author = {Chunpeng Zhou and Zhi Yu and Xilu Yuan and Sheng Zhou and Jiajun Bu and Haishuai Wang},
  title = {{Less is more: A closer look at semantic-based few-shot learning}},
  journal = {Information Fusion},
  year = {2025},
  doi = {10.1016/j.inffus.2024.102672},
}
Figure