Abstract
Vision-Language Models (VLMs) have demonstrated remarkable capabilities but face challenges when adapting to specialized downstream tasks. Traditional tuning methods often rely on a text encoder, which can be computationally heavy and limited by a fixed vocabulary. This paper introduces Concept Embedding Search (ConES), a parameter-efficient tuning method that bypasses the text encoder by optimizing "concept embeddings" directly in the joint latent space. ConES significantly reduces training time—for instance, from one hour to ten minutes for image generation—while achieving superior performance across 24 datasets, particularly in specialized domains like medical imaging.
Introduction: The Bottleneck of Text-Based Tuning
Large VLMs like CLIP and GLIP are powerful generalists but difficult to specialize. Existing adaptation strategies typically fall into three categories: manual prompt engineering, full fine-tuning, or "soft" prompt tuning. The authors identify a critical bottleneck in soft prompt tuning: the continued reliance on a pre-trained text encoder. This dependency introduces a "modality gap" because general-purpose text encoders often lack the vocabulary to represent highly specific visual concepts, such as rare medical pathologies or niche artistic styles.
Methodology: Concept Embedding Search
ConES employs a two-stage approach to bridge the vision-language gap more effectively by removing the text encoder from the tuning loop.
1. Concept Embedding Search Stage
Instead of optimizing input tokens for a text encoder, ConES initializes random vectors (concept embeddings) directly in the output space of the text encoder. By discarding the text encoder, the model optimizes these vectors using task-specific losses—such as reconstruction loss for generation or bounding-box loss for detection—while the image encoder remains frozen. This allows for a "pure" mathematical representation of visual concepts unconstrained by linguistic labels. This architectural shift simplifies the computational graph, speeding up training and preventing the "language prior" of the text encoder from biasing the learned representation.
2. Performance Boosting Stage
After identifying the optimal concept embeddings, they are fixed as prompts. The rest of the VLM is then unfrozen and fine-tuned to maximize performance on the target task. This stage ensures that the model's internal weights are perfectly aligned with the newly discovered concept embeddings.
Experimental Evaluation
The researchers validated ConES across a massive suite of 24 datasets spanning three major computer vision tasks:
- Object Detection: 13 "Object Detection in the Wild" (ODinW) datasets and 8 medical datasets (e.g., TBX11k for tuberculosis detection and LUNA16 for lung nodules).
- Instance Segmentation: Cityscapes for natural scenes and medical sets like Kavsir-SEG.
- Personalized Image Generation: The DreamBooth dataset, featuring 30 distinct subjects including pets and household objects.
Key Results and Insights
ConES outperformed standard prompt tuning and textual inversion on 21 out of 24 datasets. The performance gain was most pronounced in the medical domain, with a ~5% increase in Average Precision (AP) in radiology tasks.
Key insights from the study include:
- Closing the Modality Gap: t-SNE visualizations confirm that ConES embeddings reside much closer to visual features than standard text-derived embeddings. This "blending" of modalities facilitates significantly better generalization on out-of-domain data.
- The "Wordless" Advantage: By bypassing linguistic labels, ConES captures the visual essence of complex pathologies that lack precise descriptors in general-purpose vocabularies.
- Efficiency and Detail: Removing the text encoder halves the active parameters during tuning. In image generation, ConES achieved high-quality results in just 10 minutes, compared to the hour required by traditional textual inversion. Furthermore, ConES proved superior at preserving fine-grained details, such as the specific texture of a vase or the unique geometry of sunglasses.
Conclusion and Future Work
ConES offers a lightweight and effective alternative for VLM adaptation by directly searching the latent space. While it excels in capturing visual details and improving efficiency, the authors acknowledge that future work will focus on further enhancing the aesthetic quality of generated images and extending the method to video understanding and complex visual reasoning tasks.
Summary updated