ConES: Concept Embedding Search for Parameter Efficient Tuning Large Vision Language Models

Huahui Yi, Ziyuan Qin, Wei Xu, Miaotian Guo, Kun Wang, Shaoting Zhang, Kang Li, Qicheng LaoarXiv 2023 · DOI · arXiv

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable capabilities but face challenges when adapting to specialized downstream tasks. Traditional tuning methods often rely on a text encoder, which can be computationally heavy and limited by a fixed vocabulary. This paper introduces Concept Embedding Search (ConES), a parameter-efficient tuning method that bypasses the text encoder by optimizing "concept embeddings" directly in the joint latent space. ConES significantly reduces training time—for instance, from one hour to ten minutes for image generation—while achieving superior performance across 24 datasets, particularly in specialized domains like medical imaging.

Introduction: The Bottleneck of Text-Based Tuning

Large VLMs like CLIP and GLIP are powerful generalists but difficult to specialize. Existing adaptation strategies typically fall into three categories: manual prompt engineering, full fine-tuning, or "soft" prompt tuning. The authors identify a critical bottleneck in soft prompt tuning: the continued reliance on a pre-trained text encoder. This dependency introduces a "modality gap" because general-purpose text encoders often lack the vocabulary to represent highly specific visual concepts, such as rare medical pathologies or niche artistic styles.

Comparison of ConES with existing tuning methods like Prompt Engineering and Textual Inversion.

ConES employs a two-stage approach to bridge the vision-language gap more effectively by removing the text encoder from the tuning loop.

1. Concept Embedding Search Stage

Instead of optimizing input tokens for a text encoder, ConES initializes random vectors (concept embeddings) directly in the output space of the text encoder. By discarding the text encoder, the model optimizes these vectors using task-specific losses—such as reconstruction loss for generation or bounding-box loss for detection—while the image encoder remains frozen. This allows for a "pure" mathematical representation of visual concepts unconstrained by linguistic labels. This architectural shift simplifies the computational graph, speeding up training and preventing the "language prior" of the text encoder from biasing the learned representation.

2. Performance Boosting Stage

After identifying the optimal concept embeddings, they are fixed as prompts. The rest of the VLM is then unfrozen and fine-tuned to maximize performance on the target task. This stage ensures that the model's internal weights are perfectly aligned with the newly discovered concept embeddings.

The two-stage ConES pipeline: Concept Embedding Search and Performance Boosting.

Experimental Evaluation

The researchers validated ConES across a massive suite of 24 datasets spanning three major computer vision tasks:

  • Object Detection: 13 "Object Detection in the Wild" (ODinW) datasets and 8 medical datasets (e.g., TBX11k for tuberculosis detection and LUNA16 for lung nodules).
  • Instance Segmentation: Cityscapes for natural scenes and medical sets like Kavsir-SEG.
  • Personalized Image Generation: The DreamBooth dataset, featuring 30 distinct subjects including pets and household objects.

Key Results and Insights

ConES outperformed standard prompt tuning and textual inversion on 21 out of 24 datasets. The performance gain was most pronounced in the medical domain, with a ~5% increase in Average Precision (AP) in radiology tasks.

Key insights from the study include:

  • Closing the Modality Gap: t-SNE visualizations confirm that ConES embeddings reside much closer to visual features than standard text-derived embeddings. This "blending" of modalities facilitates significantly better generalization on out-of-domain data.
  • The "Wordless" Advantage: By bypassing linguistic labels, ConES captures the visual essence of complex pathologies that lack precise descriptors in general-purpose vocabularies.
  • Efficiency and Detail: Removing the text encoder halves the active parameters during tuning. In image generation, ConES achieved high-quality results in just 10 minutes, compared to the hour required by traditional textual inversion. Furthermore, ConES proved superior at preserving fine-grained details, such as the specific texture of a vase or the unique geometry of sunglasses.

Conclusion and Future Work

ConES offers a lightweight and effective alternative for VLM adaptation by directly searching the latent space. While it excels in capturing visual details and improving efficiency, the authors acknowledge that future work will focus on further enhancing the aesthetic quality of generated images and extending the method to video understanding and complex visual reasoning tasks.

Summary updated

Abstract

Large pre-trained vision-language models have shown great prominence in transferring pre-acquired knowledge to various domains and downstream tasks with appropriate prompting or tuning. Existing prevalent tuning methods can be generally categorized into three genres: 1) prompt engineering by creating suitable prompt texts, which is time-consuming and requires domain expertise; 2) or simply fine-tuning the whole model, which is extremely inefficient; 3) prompt tuning through parameterized prompt embeddings with the text encoder. Nevertheless, all methods rely on the text encoder for bridging the modality gap between vision and language. In this work, we question the necessity of the cumbersome text encoder for a more lightweight and efficient tuning paradigm as well as more representative prompt embeddings closer to the image representations. To achieve this, we propose a Concept Embedding Search (ConES) approach by optimizing prompt embeddings -- without the need of the text encoder -- to capture the 'concept' of the image modality through a variety of task objectives. By dropping the text encoder, we are able to significantly speed up the learning process, \eg, from about an hour to just ten minutes in our experiments for personalized text-to-image generation without impairing the generation quality. Moreover, our proposed approach is orthogonal to current existing tuning methods since the searched concept embeddings can be further utilized in the next stage of fine-tuning the pre-trained large models for boosting performance. Extensive experiments show that our approach can beat the prompt tuning and textual inversion methods in a variety of downstream tasks including objection detection, instance segmentation, and image generation. Our approach also shows better generalization capability for unseen concepts in specialized domains, such as the medical domain.
BibTeX
@misc{yi2023cones,
  author = {Huahui Yi and Ziyuan Qin and Wei Xu and Miaotian Guo and Kun Wang and Shaoting Zhang and Kang Li and Qicheng Lao},
  title = {{ConES: Concept Embedding Search for Parameter Efficient Tuning Large Vision Language Models}},
  year = {2023},
  eprint = {2305.18993},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.18993},
}
Figure