Large vision-language models (VLMs) like CLIP, SAM2, and LLaVA possess an extraordinary grasp of the visual and textual world. Because they are trained on massive, diverse datasets, they construct a highly optimized, high-dimensional latent space where images and text are deeply aligned.
However, when we want to deploy these generalists to specialized downstream tasks—such as detecting deepfakes, segmenting specific medical anomalies, or understanding 3D point clouds—we face a fundamental dilemma. Full fine-tuning (updating every weight in the network) is not only computationally prohibitive, but it also warps the model's pre-trained manifold, causing it to forget its broad, zero-shot capabilities.
Instead of rewriting the entire network, a powerful paradigm has emerged: frozen backbone adaptation. By keeping the core parameters of the foundation model completely static, researchers introduce tiny, targeted, learnable modules. This approach builds on a compelling intuition: the foundation model already knows enough to solve the task; we just need to learn how to steer its existing representations.
Input-Space Steering via Prompt Tuning
The least invasive way to adapt a frozen model is to modify its inputs rather than its internal mechanics. In text-heavy or multi-modal foundation models, this is achieved by appending learnable, continuous vectors—called "soft prompts"—to the input embeddings.
[Static Image] -------> [ Frozen Vision Encoder ] -------\
===> [Task Prediction]
[Learnable Prompts] --> [ Frozen Text Encoder ] -------/
Instead of relying on humans to engineer the perfect text prompt (like "a photo of a deepfake"), we let gradient descent optimize a small set of task-specific prompt tokens. For instance, You-Ming Chang et al. demonstrate in AntifakePrompt: Prompt-Tuned Vision-Language Models are Fake Image Detectors that framing deepfake detection as a visual question-answering task and optimizing only the soft prompt embeddings yields state-of-the-art accuracy on unseen generative models while using a fraction of the data.
Prompt tuning works because it shifts the context in the joint latent space. Rather than forcing the model to learn new visual features, the optimized prompts act as a directional beacon, guiding the frozen model to look at the image through a specific task lens. This can be made even more precise by structuring the prompts hierarchically. Tong Ding et al. show in Tree of Attributes Prompt Learning for Vision-Language Models that using an LLM to generate a structured tree of visual attributes, and optimizing "expert" tokens for those attributes, dramatically improves how well the frozen model generalizes to entirely new classes.
Alternatively, instead of prepending tokens, we can apply a subtle shift directly to the final embeddings. Ruoyu Feng et al. introduce this in Rethinking Domain Adaptation and Generalization in the Era of CLIP via text-embedding residual tuning. By adding a small learnable vector directly to the fixed text embeddings, they capture domain-specific nuances without altering the underlying text encoder.
Feature-Space Refinement via Lightweight Adapters
While input prompting steers the model globally, some tasks require structural or architectural bridges—such as mapping a new modality into the VLM's feature space or performing dense, pixel-level predictions. In these scenarios, researchers insert lightweight, trainable bottleneck layers (adapters) or residual branches alongside the frozen layers.
Consider the challenge of adding language understanding to a model that only understands vision. The Segment Anything Model v2 (SAM2) is exceptional at carving out masks from visual clicks, but natively lacks a text interface. To bridge this gap, Shiting Xiao et al. developed OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts. By connecting a frozen vision-language model to the frozen SAM2 backbone via a lightweight adapter, they unlock open-vocabulary text-to-mask segmentation with minimal parameter updates.
[Frozen VLM] -------> [ Lightweight Adapter ] -------> [ Frozen SAM2 ] ---> [Mask]
Adapters also excel at cross-modal alignment, such as translating 3D data into an existing 2D vision-language space. Ye Mao et al. demonstrate in OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images that adapting a pre-trained VLM using a small residual parameter branch is far more effective than training a massive 3D encoder from scratch. The residual branch simply learns to project depth-aligned image features into the VLM's established manifold.
When dealing with environmental shifts, adapters can be isolated to handle distinct variations. Haochen Li introduces a dual-adapter strategy in DA-Ada: Learning Domain-Aware Adapter for Domain Adaptive Object Detection, utilizing a Domain-Invariant Adapter to find common ground between environments and a Domain-Specific Adapter to recover localized target details, all while keeping the primary VLM frozen. Similarly, for sequential learning, Julien Nicolas shows in MoP-CLIP: A Mixture of Prompt-Tuned CLIP Models for Domain Incremental Learning that routing features through a mixture of specialized, domain-specific prompt modules completely prevents catastrophic forgetting during incremental tasks.
Strategic Asymmetry and Granularity Control
Not all layers in a deep neural network are created equal. Shallow layers typically capture low-level geometric features (edges, textures), while deeper layers capture high-level semantic concepts. Recognizing this asymmetry allows for even more aggressive optimization.
In Skip Tuning: Pre-trained Vision-Language Models are Effective and Efficient Adapters Themselves, Shihan Wu et al. exploit this property by caching the activations of shallow layers and only updating the deeper layers during adaptation. Because the low-level visual representations remain constant across most downstream tasks, bypassing backpropagation through the early layers slashes computational overhead while preserving task-relevant semantic flexibility.
A similar principle applies to the functional roles of encoders versus decoders. When adapting generative vision-language models, updating the components symmetrically can degrade the model's natural language fluency. To combat this, Yanyang Li et al. introduce the concept of granularity control in VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Control.
[Visual Input] ---> [ Encoder (Heavier Updates) ] ---\
===> [ Task Output ]
[Textual Input] ---> [ Decoder (Lightweight/Frozen) ] -/
By applying heavier parameter updates to the encoders (to align the new visual distributions) while keeping the decoder modifications extremely lightweight, the system ensures that the model retains its generative linguistic capabilities while rapidly mastering new multi-modal tasks.
Why It Works: The Geometric Intuition
To understand why frozen backbone adaptation is so effective, it helps to visualize the model's pre-trained representation space as a highly complex, curved manifold.
When we fully fine-tune a model on a small downstream dataset, the massive influx of gradients acts like a sledgehammer, warping and collapsing the global geometry of this manifold to satisfy a narrow local objective.
Parameter-efficient adaptation acts instead like a set of precise local lenses or coordinate transformations. Prompt tuning shifts the starting position of our queries on the manifold, ensuring they land in a region where the frozen model's existing knowledge is highly relevant. Adapters act as localized residual corrections, performing minor warping at critical junctions to align new tasks or modalities without destroying the global structure.
By respecting the geometric integrity of the foundation model's pre-trained space, frozen backbone adaptation achieves a rare trifecta in deep learning: drastically reduced training costs, robust defense against catastrophic forgetting, and state-of-the-art performance on specialized frontiers.
Key Foundational Work
ClipCap: CLIP Prefix for Image Captioning (2021) — contributed 1 concept elements to this direction.