When deep learning architectures converged on the Transformer block, it sparked a parallel revolution in model adaptation. Instead of fine-tuning billions of parameters for every new task, researchers developed Parameter-Efficient Fine-Tuning (PEFT) techniques—such as low-rank adaptation (LoRA), adapters, and prompt tuning—to update only a tiny fraction of a model's weights.
However, these adaptation strategies were largely engineered within unimodal silos. A method designed to exploit the structural redundancies of natural language text does not necessarily align with the spatial dependencies of computer vision or the heterogeneous demands of unified vision-language models. The core challenge of cross-modal architectural transferability is understanding why an adaptation mechanism succeeds in one modality and determining whether its underlying structural assumptions hold true when crossing modal boundaries.
Modality Asymmetries and Structural Assumptions
To understand why adaptation strategies do not always transfer seamlessly across modalities, we must look at how different architectures distribute and store knowledge across their layers. PEFT methods work by exploiting specific structural properties of a pre-trained model, but these properties vary drastically between vision and language networks.
For example, low-rank adaptation assumes that weight updates during adaptation occupy a subspace with a low intrinsic dimension. In Low-Rank Rescaled Vision Transformer Fine-Tuning: A Residual Design Approach, Wei Dong et al. leverage this by introducing a residual-based low-rank rescaling method tailored specifically for Vision Transformers (ViTs). Yet, the authors note that transferring such low-rank rescaling techniques directly to natural language processing remains an open question due to fundamental differences in attention distributions and data characteristics between spatial vision tokens and sequential text tokens.
Furthermore, the vertical distribution of knowledge—which layers learn what—is highly modality-dependent:
- In Language Models: Shallow layers appear to be the primary engines for absorbing new factual or structural information. Tianxiang Chen et al. demonstrate in Llama SLayer 8B: Shallow Layers Hold the Key to Knowledge Injection that selectively expanding and adapting the shallow layers of Large Language Models (LLMs) yields far better knowledge injection than modifying deeper layers.
- In Vision-Language Models: The localization shifts. When adapting contrastive models like CLIP to new visual domains, Wenxuan Zhang et al. show in Overcoming Generic Knowledge Loss with Selective Parameter Update that restricting updates to the first few multi-layer perceptron (MLP) layers is what preserves the model's broad, pre-trained generic knowledge while maximizing downstream task accuracy.
Because of these asymmetries, an optimization or pruning strategy that assumes a specific pattern of layer redundancy in an LLM may fail entirely when applied to a ViT or a joint multi-modal network.
The Friction of Multi-Module Integration
The transferability problem compounds when moving from single-modality models to unified, multimodal architectures (such as Vision-Language Models). In these systems, parameter-efficient adapters must not only adapt individual backbones but also bridge the representation gap between entirely different data streams.
As Xiongtao Zhou et al. observe in An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models, applying PEFT to multimodal architectures introduces a complex "black box" interaction. Traditional PEFT research focuses almost exclusively on text-only sequences, leaving a significant gap in our understanding of how these adapters interact with non-textual components like visual encoders and the alignment connectors (such as projection layers or Q-Formers) that link them.
Unimodal Sequence:
[Text Tokens] ------> [PEFT Adapter] ------> [Text Output] (Predictable Dynamics)
Multimodal Sequence:
[Visual Tokens] ---\
+--> [Alignment Connector] ------> [PEFT Adapter] ------> [Multimodal Output]
[Text Tokens] -----/ ^
|
(Unpredictable Cross-Modal Friction)
This friction is precisely what makes direct transfer difficult. In Prompt Tuning for Unified Multimodal Pretrained Models, Hao Yang et al. investigate whether parameter-efficient methods can successfully bridge the gap between disparate modalities within a single sequence-to-sequence framework without updating the entire model backbone. The challenge is that a prompt or an adapter must learn to modulate the cross-attention dynamics between vision and language tokens simultaneously, a significantly more intricate optimization landscape than standard unimodal steering.
Emerging Paradigms for Cross-Modal Modularity
To resolve these architectural mismatches, recent research has shifted away from monolithic adapter deployment toward dynamic, modular routing and training-free weight composition. Instead of forcing a single adapter architecture to fit all modalities, these strategies compose or merge specialized modules on the fly.
Modular Routing and Skill Inventories
One promising direction is to decouple task-specific and modality-specific abilities into an inventory of independent modules, using a routing function to combine them. Edoardo Ponti et al. introduce this concept in Combining Parameter-efficient Modules for Task-level Generalisation via the SKILLED model, which allows tasks to dynamically recruit a variable number of parameter-efficient skills.
Similarly, to handle the diverse domains encountered in visual question answering, Huishan Ji et al. propose a Mixture of LoRAs (MoL) strategy in Towards One-to-Many Visual Question Answering. By dynamically blending task-specific "ability" adapters, a single model can generalize across diverse downstream tasks. However, as the authors note, the cross-modal generalizability of such merging strategies to broader tasks like image captioning or grounding remains an active area of investigation.
Training-Free Weight Merging
When a language model is adapted to "see" via multimodal instruction tuning, it often suffers from language degradation—losing its edge in pure-text reasoning. Rather than launching expensive retraining cycles to balance the modalities, researchers have found that the architecture's weights can be manipulated directly after training.
In Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning, Neale Ratzlaff et al. demonstrate that this degradation can be mitigated without any extra training. By simply "merging" the fine-tuned multimodal model's weights back with its original text-only parent, the model recovers its lost linguistic reasoning capabilities. This indicates that cross-modal alignment does not require destroying original unimodal structures; rather, the specialized knowledge can be mathematically integrated back into the base architecture.
Ultimately, achieving true cross-modal architectural transferability requires moving past empirical trial-and-error. By mapping how different modalities distribute information across layers and understanding how adapters interact with cross-modal connectors, the field is moving toward unified foundation models that can be adapted efficiently, regardless of the data stream they process.