When adapting a pre-trained vision-language model (VLM) like CLIP to a new task, the most common approach is to keep the majority of the model frozen and tune a small set of parameters. However, a recurring observation across recent research is that this adaptation often suffers from unimodal adaptation bias: a bottleneck that occurs when only one side of the model—either the visual encoder or the textual encoder—is modified.
As Liang et al. argue, restricting adaptation to a single modality prevents the model from fully resolving discrepancies between visual inputs and linguistic descriptions in novel environments. This "one-sided" approach creates a representational gap that hinders the model's ability to perform complex cross-modal reasoning.
The Bottleneck of Unimodal Tuning
Most parameter-efficient fine-tuning methods, such as prompt tuning, focus exclusively on the text branch. While this is computationally cheap, Qiu et al. note that leaving the visual encoder static limits the model's capacity to bridge the semantic gap. If the visual features of a new domain (like medical imaging or satellite photos) differ significantly from the general-purpose data the model was trained on, simply changing the text prompts is insufficient.
This asymmetry leads to several specific failures:
- Alignment Mismatch: Nishida et al. and the authors of DueT observe that freezing the image encoder while tuning only the text encoder prevents the system from aligning complex visual features with new linguistic contexts, particularly in non-English languages.
- Domain Invariance: Kan et al. point out that a one-sided approach fails to address fundamental differences in how objects are visually represented across diverse styles or environments.
- Semantic Neglect: In transductive settings, Baklouti et al. find that focusing primarily on visual features overlooks the rich semantic information provided by class descriptions, which is essential for maintaining alignment during fine-tuning.
Synergistic and Dual-Modality Adaptation
To solve this, researchers have moved toward dual-modality adaptation. The intuition is simple: for the vision and language representations to meet in the middle, both must be allowed to move.
Wang et al. demonstrate this with M²PT, which injects learnable "soft prompts" into both the vision and language processors simultaneously. By updating both sides, the model achieves superior zero-shot performance compared to full fine-tuning, despite using less than 0.1% of the parameters. Similarly, Liu et al. propose CSKT (Synergistic Knowledge Transfer), which uses dual adapters to exploit the mutual relationship between modalities, achieving state-of-the-art accuracy in person retrieval tasks.
This synergy is not just about updating both sides independently, but about ensuring they remain synchronized. Guo et al. argue that without this synchronization, models suffer from a "modality bias" where the learning process focuses disproportionately on one input type, preventing the development of a balanced, unified representation.
From Static Alignment to Dynamic Interaction
Beyond just updating parameters in both branches, a more advanced solution involves enabling the modalities to "talk" to each other during inference.
Standard models often use a unidirectional flow—for example, using text to classify an image. However, Lv et al. introduce Bi-directional Modality Interaction Prompting (BMIP), arguing that the vision modality should also inform the language modality. This is particularly useful when visual features are more descriptive or reliable than the provided text.
Other researchers have explored making these interactions dynamic:
- Visual-Guided Text: Zhang et al. developed VT-CLIP, which allows text prompts to "look" at the image via cross-attention. This creates image-specific text features rather than relying on static, global prompts.
- Iterative Refinement: Qiu et al. propose a "double-checking" mechanism where the model progressively aligns visual and textual features over multiple passes, mimicking human visual verification.
- Relational Modeling: Li et al. use a Dual Knowledge Graph to model relationships between classes in both modalities, capturing the inter-class correlations that unimodal methods ignore.
By treating vision and language as two sides of the same coin—as Kwon et al. do in their masked reconstruction framework—these methods move away from the "toothpick steering a cargo ship" problem described by Li. Instead of forcing one modality to do all the heavy lifting, dual-modality adaptation allows the model to maintain the structural relationships and semantic alignment necessary for robust generalization.
Key Foundational Work
Proto-CLIP: Vision-Language Prototypical Network for Few-Shot Learning (2023) — contributed 1 concept elements to this direction.
DueT: Image-Text Contrastive Transfer Learning with Dual-adapter Tuning (2023) — contributed 1 concept elements to this direction.
Towards Realistic Unsupervised Fine-tuning with CLIP (2023) — contributed 1 concept elements to this direction.