CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Values

Taha Koleilat, Hassan Rivaz, Yiming XiaoarXiv 2025 · arXiv

TL;DR

CLIP-SVD introduces a highly parameter-efficient and interpretable method for adapting Vision-Language Models (VLMs) like CLIP to new domains by fine-tuning only their singular values. By freezing the singular vectors and adjusting just 0.04% of the model’s total parameters, it achieves state-of-the-art few-shot performance and base-to-novel generalization across 21 diverse natural and biomedical datasets without adding any structural overhead.

What Problem Does This Paper Address?

Vision-Language Models (VLMs) like CLIP excel at zero-shot learning, but adapting them to specialized, fine-grained domains (such as medical imaging) remains a significant challenge. Standard approaches like full fine-tuning are computationally prohibitive.

To solve this, Parameter-Efficient Fine-Tuning (PEFT) methods like prompt tuning (e.g., CoOp) or adapter modules (e.g., MaPLe, LoRA) introduce external, randomly initialized components. However, these "add-on" structures can destabilize the model, increase inference latency, and cause the model to forget the rich, generalized knowledge it learned during pretraining. Furthermore, understanding how these models change internally during adaptation has historically been a black box.

Methods

The authors introduce CLIP-SVD, which applies Singular Value Fine-Tuning (SVF) to both the vision and text branches of Transformer-based VLMs.

Instead of adding new parameters, CLIP-SVD decomposes the existing weight matrices ($W$) of the Multi-Head Self-Attention (MHSA) and Multi-Layer Perceptron (MLP) blocks using Singular Value Decomposition (SVD):

$$W = USR^\top$$

Where $U$ and $R$ are the left and right singular vectors (representing the geometric basis directions of the parameter space), and $S$ is a diagonal matrix containing the singular values (representing the magnitude or importance of those directions).

During adaptation, $U$ and $R$ are kept completely frozen to preserve the model's pretrained structural geometry. Only the singular values in $S$ are fine-tuned. This allows the model to rescale its intrinsic features to fit the new domain using an incredibly small parameter budget.

Figure 1: The overall framework of CLIP-SVD. We decompose the Query, Key, Value, and Output projection weights (WQ, WK, WV and WO) of the MHSA blocks in both vision and text encoders Ev and Et, as well as the linear weights of the Feed-forward Networks (WMLP ) in all layers. We finetune only the singular values S of the SVD decomposed weights.

Additionally, the authors combine CLIP-SVD with an interpretability framework called TextSpan. Because the basis vectors ($U$) are frozen, the semantic directions of the attention heads remain constant. By tracking which singular values change the most, the authors use a large language model (LLM) to generate natural language descriptions of exactly what features (e.g., "contour irregularity", "aerial landscapes") the model is prioritizing after adaptation.

Datasets

The framework was rigorously evaluated across two distinct domains:

  • Natural Image Domain (11 datasets): ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, and UCF101.
  • Biomedical Domain (10 datasets): Covering 9 organs and 8 modalities, including CTKidney, Kvasir (endoscopy), RETINA, LC25000 & CHMNIST (histopathology), BTMRI (brain MRI), OCTMNIST, BUSI (breast ultrasound), COVID-QU-Ex (chest X-ray), and KneeXray.
  • Out-of-Distribution Generalization: ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.

To facilitate biomedical interpretability, the authors also curated and validated a new biomedical corpus of 300 descriptive prompts detailing clinical visual features like contrast, shape, and texture.

Key Insights and Ideas

  • Preserving the Subspace is Key: Traditional PEFT methods like LoRA introduce new degrees of freedom that can distort the original model geometry. CLIP-SVD proves that simply rescaling the existing pretrained subspace is sufficient—and often superior—for few-shot adaptation.
  • Multi-Modal Tuning Matters: Fine-tuning singular values in both the text and vision encoders simultaneously yields better results than tuning either modality alone, demonstrating that cross-modal alignment can be optimized via weight magnitudes.
  • Built-in Interpretability: Because the singular vectors are frozen, the semantic meaning of individual attention heads doesn't drift. This unique property allows researchers to "read" the model's adaptation process in plain English, transforming post-hoc interpretability into an inherent feature of the fine-tuning method.
Table 1: Natural language-based interpretations for the top 3 Attention Heads associated with the highest normalized changes (sorted in descending order) in the Output-Value circuit after CLIP adaptation for different datasets. Here, “L” denotes layer while “H” denotes attention head.

Key Results

  • Parameter Efficiency: CLIP-SVD requires updating only 0.04% of the model's total parameters (92.2K for CLIP; 110.6K for BiomedCLIP), which is significantly lower than methods like MaPLe (3.5M parameters).
  • Natural Domain Performance: In few-shot classification, CLIP-SVD consistently outperformed competitive baselines, achieving a 73.20% average accuracy in the 1-shot setting compared to CLIP-LoRA's 72.20%.
  • Biomedical Domain Performance: The performance leap was even more pronounced in the challenging medical domain, where CLIP-SVD outperformed the second-best method (BiomedCoOp) by +4.28% in the 8-shot setting (73.24% vs. 68.96%).
  • Generalization: In base-to-novel generalization tests, CLIP-SVD achieved the highest harmonic mean accuracy across both natural (80.13%) and biomedical (78.25%) benchmarks, proving it retains zero-shot capabilities better than structural adapters.

Limitations and Future Work

While highly effective in low-data (few-shot) regimes, freezing the singular vectors ($U$ and $R$) inherently limits the total expressivity of the parameter space. In fully supervised, data-rich scenarios, this constraint might prevent the model from learning entirely new visual features that weren't present during pretraining.

Additionally, the method requires a one-time SVD preprocessing step at initialization. While computationally modest (~12 to 15 seconds), it is an extra step not required by randomly initialized adapters. Future work includes validating the natural language interpretations deeper with domain experts and exploring applications in broader multi-modal architectures.

Summary updated

Abstract

Vision-language models (VLMs) like CLIP have shown impressive zero-shot and few-shot learning capabilities across diverse applications. However, adapting these models to new fine-grained domains remains difficult due to reliance on prompt engineering and the high cost of full model fine-tuning. Existing adaptation approaches rely on augmented components, such as prompt tokens and adapter modules, which could limit adaptation quality, destabilize the model, and compromise the rich knowledge learned during pretraining. In this work, we present CLIP-SVD, a multi-modal and parameter-efficient adaptation framework that applies Singular Value Fine-tuning (SVF) to CLIP, leveraging Singular Value Decomposition (SVD) to modify the internal parameter space of CLIP without injecting additional modules. Specifically, we fine-tune only the singular values of the CLIP parameter matrices to rescale the basis vectors for domain adaptation while retaining the pretrained model. This design enables enhanced adaptation performance using only 0.04% of the model's total parameters and better preservation of its generalization ability. CLIP-SVD achieves state-of-the-art classification results on 11 natural and 10 biomedical datasets, outperforming previous methods in both accuracy and generalization under few-shot settings. Additionally, we leverage a natural language-based approach to analyze the effectiveness and dynamics of the CLIP adaptation to allow interpretability of CLIP-SVD. Overall, this work provides the first extensive empirical evaluation of SVD-based finetuning in the vision-language model setting. The code and biomedical corpus are publicly available at https://github.com/HealthX-Lab/CLIP-SVD.
BibTeX
@misc{koleilat2025clip,
  author = {Taha Koleilat and Hassan Rivaz and Yiming Xiao},
  title = {{CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Values}},
  year = {2025},
  eprint = {2509.03740},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.03740},
}
Figure