FewVS: A Vision-Semantics Integration Framework for Few-Shot Image Classification

Zhuoling Li, Yong Wang, Kaitong Li

Abstract

Some recent methods address few-shot image classification by extracting semantic information from class names and devising mechanisms for aligning vision and semantics to integrate information from both modalities. However, class names provide only abstract information, which is insufficient to capture the visual details present in images. As a result, this vision-semantics alignment is inherently biased, leading to sub-optimal integration outcomes. In this paper, we avoid this biased vision-semantics alignment by introducing CLIP, a natural bridge between vision and semantics, and enforcing unbiased vision-vision alignment as a proxy task. Specifically, we align features encoded from the same image by both the few-shot encoder and CLIP's vision encoder. This alignment is accomplished through a linear layer, with a training objective formulated using optimal transport-based assignment prediction. Thanks to the inherent alignment between CLIP's vision and text encoders, the few-shot encoder is indirectly aligned to CLIP's text encoder, which serves as the foundation for better vision-semantics integration. In addition, to further improve vision-semantics integration at the testing stage, we mine potential fine-grained semantic attributes of class names from large language models. Correspondingly, an online optimization module is designed to adaptively integrate the semantic attributes and visual information extracted from images. Extensive results on four datasets demonstrate that our method outperforms state-of-the-art methods.
BibTeX
@inproceedings{li2024fewvs,
  author = {Zhuoling Li and Yong Wang and Kaitong Li},
  title = {{FewVS: A Vision-Semantics Integration Framework for Few-Shot Image Classification}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3681427},
}

Loading related content...

Figure