VT-FSL: Bridging Vision and Text with LLMs for Few-Shot Learning

Wenhao Li, Qiangchang Wang, Xianjing Meng, Zhibin Wu, Yilong Yin

Introduction

Few-Shot Learning (FSL) represents a critical frontier in computer vision, aiming to enable models to recognize novel categories from a minimal number of labeled examples. While traditional approaches have increasingly relied on "semantic guidance"—leveraging class names or textual descriptions to bridge the data gap—they often suffer from two primary limitations: semantic hallucination and suboptimal multi-modal alignment.

Semantic hallucination occurs when Large Language Models (LLMs) generate generic descriptions based solely on class names, often including features not present in the specific visual samples provided. Furthermore, standard alignment techniques typically rely on pairwise contrastive learning, which can fail to capture the complex, non-linear relationships between multiple data modalities simultaneously. This paper introduces VT-FSL, a framework designed to overcome these hurdles by grounding textual descriptions in visual evidence and employing a novel geometric alignment strategy.

Methodology

VT-FSL introduces two core modules to synchronize vision and text: Cross-modal Iterative Prompting (CIP) and Cross-modal Geometric Alignment (CGA).

Cross-modal Iterative Prompting (CIP)

To ensure that textual descriptions are relevant to the available support images, the CIP module treats the LLM as a "visual taxonomy expert." Instead of relying on class labels alone, CIP provides the LLM with both the label and the few available support images. The module follows a structured four-stage reasoning process:

  1. Strategy: Determining the key discriminative features of the class.
  2. Perception: Analyzing the provided support images for these features.
  3. Refinement: Filtering out generic or non-present traits.
  4. Conclusion: Generating a visually grounded description.

This grounded description is subsequently used by a generative model (such as DALL-E 3) to synthesize additional visual samples. Because these synthetic images are derived from grounded text, they are significantly less prone to the noise and hallucinations that plague standard generative augmentation.

Cross-modal Geometric Alignment (CGA)

The CGA module addresses the limitations of pairwise alignment. Rather than simply comparing two vectors, CGA treats the support image, the generated text, and the synthetic image as three vectors in a Reproducing Kernel Hilbert Space (RKHS). It calculates the volume of a 3-dimensional parallelotope spanned by these three vectors. By minimizing this volume, the model forces all three modalities into a tight, consistent cluster. This geometric approach captures global relationships and non-linear dependencies that traditional distance-based metrics often overlook.

Experimental Evaluation

The researchers benchmarked VT-FSL across ten diverse datasets to evaluate its robustness and versatility:

  • General FSL: miniImageNet, tieredImageNet, CIFAR-FS, and FC100.
  • Fine-grained FSL: CUB-200-2011, Stanford-Dogs, and Stanford-Cars.
  • Cross-domain FSL: Training on miniImageNet and testing on disparate domains such as Places and Plantae.

Key Results

VT-FSL established a new state-of-the-art (SOTA) across all ten benchmarks. On the miniImageNet 1-shot task, the model achieved an accuracy of 83.66%, representing a substantial improvement over previous leading methods like SemFew (78.94%) and ECER (81.14%).

The framework's performance in cross-domain scenarios was particularly noteworthy, showing improvements of up to 15% over certain baselines. This suggests that grounded semantic descriptions are highly transferable, even when the visual domain shifts significantly. Furthermore, despite the use of sophisticated LLMs during the setup phase, the model remains highly efficient. Because the LLM and image generation occur in an offline stage, the final model achieves an inference time of just 76ms.

Discussion and Limitations

VT-FSL demonstrates that grounding LLMs in visual data is essential for effective semantic guidance. By shifting from "dictionary-style" definitions to "visual taxonomy," the model avoids the pitfalls of generic textual noise. Additionally, the transition from distance-based alignment to volume-based alignment provides a more holistic way to integrate multiple data sources.

However, some limitations remain. The quality of the framework is inherently tied to the capabilities of the underlying generative and language models; poor synthetic outputs can still introduce misleading signals. Furthermore, while the cross-domain results are strong, the authors note that the current benchmarks still share some fundamental visual characteristics. Future work is needed to explore the model's performance in extreme domain shifts, such as transitioning from natural imagery to specialized fields like medical radiography.

Conclusion

VT-FSL provides a robust framework for few-shot learning by effectively bridging the gap between vision and language. Through iterative prompting and geometric alignment, it ensures that semantic information is both visually relevant and mathematically integrated, setting a new benchmark for the field.

Summary updated

BibTeX
@inproceedings{li2025vt,
  author = {Wenhao Li and Qiangchang Wang and Xianjing Meng and Zhibin Wu and Yilong Yin},
  title = {{VT-FSL: Bridging Vision and Text with LLMs for Few-Shot Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=V8sDJ1oMEy},
}
Figure