Preserving Pretrained Capabilities During Multimodal Adaptation

76 concepts64 papers

When a Large Language Model (LLM) undergoes fine-tuning to process a new sensory modality—such as vision or speech—it frequently suffers from a severe degradation of its original capabilities. An LLM that previously demonstrated sophisticated logical reasoning, summarization, and encyclopedic recall can suddenly lose its linguistic proficiency or fail standard text-only benchmarks.

This phenomenon, widely termed cross-modal catastrophic forgetting, presents a fundamental barrier to creating unified, general-purpose intelligent agents. As surveyed by Jiayang Wu et al. in Multimodal Large Language Models: A Survey, the tension between specialized perceptual training and the maintenance of foundational reasoning is a central challenge in modern artificial intelligence.

The Mechanics of Cross-Modal Forgetting

To understand why cross-modal forgetting occurs, consider what happens to a model's internal representations during multimodal fine-tuning. When forced to map high-dimensional continuous audio or visual features into a discrete textual space, the optimization process shifts the model's internal weights toward modality-specific features. Qingkai Fang et al. observe in LLaMA-Omni 2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesis that this representation shift directly compromises general-purpose intelligence at the expense of sensory alignment.

Furthermore, this degradation is driven by two distinct factors: shifts in the input modality and shifts in the required task format. As formalized by Weiguo Pian et al. in Modality-Inconsistent Continual Learning of Multimodal Large Language Models, a model can retain the ability to process a specific modality while completely forgetting the specialized reasoning or formatting required for a task type (such as transitioning from captioning to question answering).

In extreme cases, training exclusively on narrow tasks causes severe behavioral collapse:

Token Bias vs. Permanent Erasure

An insightful nuance discovered by Yaoyao Liu et al. in How to Teach Large Multimodal Models New Skills is that much of this "forgetting" is actually a temporary shift in output token bias rather than a permanent erasure of knowledge. By selectively targeting self-attention projections or specific MLP layers during tuning, researchers can introduce niche skills without disrupting the model's core general-purpose capabilities. Conversely, naive interventions like direct codebook replacement, as explored by Bohan Zhou in UniCode: Learning a Unified Codebook for Multimodal Large Language Models, introduce severe representation instability that destroys pre-trained linguistic capabilities.

Architectural Isolation: Keeping the Core Frozen

The most direct way to prevent cross-modal forgetting is to leave the pre-trained language model completely untouched. In their seminal work Multimodal Few-Shot Learning with Frozen Language Models, Maria Tsimpoukelli et al. introduced "Frozen," a method that trains an external vision encoder to translate images into the continuous word-embedding space of a frozen language model. Because the LLM's weights are immutable, its encyclopedic knowledge and linguistic patterns remain perfectly preserved.

This philosophy extends to downstream tasks like visual grounding. Size Wu et al. demonstrate in F-LMM: Grounding Frozen Large Multimodal Models that extracting and refining internal attention maps from a frozen model avoids the conversational degradation that plagues fully fine-tuned alternatives.

For speech integration, a major hurdle is the length mismatch between audio and text tokens. To bypass resource-intensive speech instruction tuning, Ruchao Fan et al. developed AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLM, an adapter that uses Connectionist Temporal Classification (CTC) alignment to compress speech tokens to match text token lengths while keeping the LLM frozen. Similarly, the authors of SSR: Alignment-Aware Modality Connector for Speech Language Models use speech-text alignments to segment audio into a 1:1 mapping with text tokens, preventing text-intelligence degradation.

Routing with Mixture-of-Experts

When freezing the entire core model limits performance, researchers turn to conditional computation via Mixture-of-Experts (MoE) architectures to isolate modality-specific updates. By routing tokens through specialized sub-networks, the model can acquire new perceptual skills without overwriting general reasoning pathways.

                  [ Input Token ]
                         │
                         ▼
                [ Gating Network ]
                 ╱              ╲
                ╱                ╲
               ▼                  ▼
      [ Visual Expert ]   [ Language Expert ]
      (Modality-Specific)   (Frozen/Preserved)
               ╲                  ╱
                ╲                ╱
                 ▼              ▼
                 [ Output Token ]

In Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation, Maohao Shen et al. utilize a Mixture-of-LoRA Experts (MoE) architecture to achieve state-of-the-art text-to-speech synthesis while successfully avoiding the degradation of text-based reasoning. This structural separation is also effective for vision; Zhengzhuo Xu et al. introduce ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understanding, which uses specialized connector experts to master chart interpretation while fully retaining the model's ability to understand general, non-chart images.

Data Blending and Interleaved Pre-training

If weights must be modified globally, the training data distribution itself must be carefully curated to anchor the model's original intelligence. Training exclusively on isolated image-caption pairs or narrow multimodal datasets introduces a severe distribution shift. As shown by Andrew Tao et al. in VILA: On Pre-training for Visual Language Models, re-blending text-only instructions and using interleaved image-text data are critical to ensuring the underlying LLM does not lose its complex reasoning capabilities.

In the audio-visual domain, sequential fine-tuning protocols (e.g., training on audio before adding video) often degrade cross-modal capabilities. To combat this, Zhengyang Li et al. demonstrate in Interleaved Audio/Audiovisual Transfer Learning for AV-ASR in Low-Resourced Languages that mixing audio-only and audiovisual data at the minibatch level prevents the model from forgetting how to utilize secondary modalities like visual lip movements.

Finally, quantization introduces unique challenges. Jingjing Xie shows in Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptation that fine-tuning quantized models exclusively on multimodal datasets causes them to lose underlying linguistic proficiency, a paradox that can be mitigated by modality-aware warmup strategies.

Key Foundational Work

Unsupervised Natural Language Inference via Decoupled Multimodal Contrastive Learning (2020) — contributed 1 concept elements to this direction.
Multimodal Few-Shot Learning with Frozen Language Models (2021) — contributed 1 concept elements to this direction.
Transitional Adaptation of Pretrained Models for Visual Storytelling (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_modal_catastrophic_forgetting_eb547685,
  title = {cross-modal catastrophic forgetting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-catastrophic-forgetting/rdr_eb5476854b2e44d5b95f37e926101267},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_modal_catastrophic_forgetting_eb547685_v1,
  title = {cross-modal catastrophic forgetting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-catastrophic-forgetting/rdr_eb5476854b2e44d5b95f37e926101267/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake