Cross-Modal Interference in Multimodal Large Language Models

52 concepts39 papers1 proposal

When we train a Large Language Model (LLM) to "see" images, "hear" audio, or interact with physical environments, we intuitively expect its overall intelligence to increase. Grounding language in sensory experiences mirrors how humans learn. However, empirical evidence across dozens of recent studies reveals a troubling paradox: integrating additional modalities often degrades a model's core linguistic capabilities, abstract reasoning, and syntactic precision.

This phenomenon, known as cross-modal interference or capacity dilution, manifests as a performance drop on pure-text tasks—such as mathematical reasoning, factual recall, and grammatical parsing—after a model undergoes multimodal training. Rather than acting as a source of cross-modal enhancement, non-textual modalities frequently behave as an architectural and optimization burden.

To build better generalist systems, we must understand why this interference occurs. The literature points to three primary mechanisms: semantic shifting toward concrete concepts, gradient conflicts during joint optimization, and structural token pollution.


The Concrete Bias: How Visual Grounding Warps Abstract Space

A text-only language model maps words into a highly optimized, high-dimensional vector space based on logical, syntactic, and abstract relationships. When we introduce visual supervision, we force this linguistic space to align with perceptual features. This alignment alters the geometry of the semantic space.

As Shaonan Wang et al. demonstrated using human neuroimaging baselines in Investigating Inner Properties of Multimodal Representation and Semantic Compositionality With Brain-Based Componential Semantics, multimodal training successfully enhances sensory-motor properties within embeddings but actively degrades the representation of abstract, social, or cognitive concepts.

Why does this happen? Visual data inherently over-indexes on concrete, physically observable entities (e.g., "dog," "red," "running"). When a model is optimized to map text to pixels, its internal representations shift to prioritize these physical correlates. Avinash Madasu and Vasudev Lal explored this trade-off in Is Multimodal Vision Supervision Beneficial to Language?, finding that text encoders trained with visual supervision consistently underperform vanilla text-only models on standard Natural Language Understanding (NLU) benchmarks. By forcing text representations to align with visual features, the model sacrifices its grasp on complex syntactic structures and purely logical reasoning that lack direct physical counterparts.

Text-Only Space:   [Abstract Logic] <---> [Grammatical Syntax] <---> [Factual Recall]
                             \                  |                  /
                              \                 v                 /
Multimodal Alignment:          v                v                v
Aligned Space:     [Concrete Nouns] <---> [Sensory-Motor Features]  (Abstract structures compressed/lost)

Gradient Conflict and Representation Disruption

The interference is not just semantic; it is also an optimization problem. During joint training or multimodal instruction tuning, a model must simultaneously satisfy unimodal objectives (like next-token text prediction) and cross-modal alignment objectives (like image-text matching).

Xinsong Zhang et al. isolated this effect in Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks. They discovered that gradients flowing from vision-language tasks often introduce noise into the unimodal encoders. Because cross-modal alignment tasks operate on less abstract, noisier perceptual signals, their gradients can overwrite or disrupt the delicate, high-level symbolic representations required for pure language understanding.

This disruption directly impacts the model's structural understanding of language. For instance, Victor Milewski and Miryam de Lhoneux evaluated how joint image-text objectives affect internal syntax in Finding Structural Knowledge in Multimodal-BERT. They found that multimodal training actively hampers the model's ability to store linguistic dependency trees, proving that broad cross-modal objectives compete with and crowd out fine-grained syntactic knowledge learned from text alone. Similarly, injecting global visual features directly into text token embeddings can immediately fracture pre-existing linguistic representations before downstream training even begins, as shown by Taichi Iki and Akiko Aizawa in Effect of Visual Extensions on Natural Language Understanding in Vision-and-Language Models.


Capacity Dilution and Token Pollution

When we move beyond vision to high-bitrate modalities like audio, or structured data like robotic constraints and database IDs, interference worsens due to architectural crowding.

In Mu2SLAM: Multitask, Multilingual Speech and Language Models, Yong Cheng et al. described this as capacity dilution. When a single fixed-capacity transformer architecture is forced to process both speech frames and text tokens, the inclusion of massive, continuous speech data dilutes the model's parameter allocation for text-based tasks.

Furthermore, forcing non-textual concepts into a shared text-decoding head introduces token pollution. For example, in video grounding tasks, models must output precise numerical timestamps alongside text. Yongxin Guo et al. demonstrated in TRACE: Temporal Grounding Video LLM via Causal Event Modeling that interleaving these numerical tokens into a shared text head causes severe interference, preventing the model from effectively leveraging its pre-trained language knowledge. A similar phenomenon occurs in recommendation systems when behavioral item identifiers are injected into an LLM's vocabulary; Reza Shirkavand et al. observed in Catalog-Native LLM: Speaking Item-ID dialect with Less Entanglement for Recommendation that this causes "knowledge interference," where behavioral recommendation patterns clash with and degrade pre-trained linguistic patterns.


Mitigating the Degradation

Recognizing that multimodal integration acts as a tax on language proficiency, researchers are developing targeted architectural and data-driven interventions to isolate and protect core linguistic skills:

  • Weight Merging and Post-Polishing: Instead of accepting language degradation as inevitable, Neale Ratzlaff et al. showed in Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning that the lost reasoning performance can be recovered without extra training. By mathematically merging the weights of the newly fine-tuned multimodal model back with its original text-only parent, they successfully retained visual comprehension while restoring text-based logic.
  • Text-Heavy Tuning Mixtures: The composition of the instruction-tuning dataset plays a critical role. The authors of MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models found that vision-centric training data actively drives language degradation. By shifting to a training mixture heavily weighted toward text (75% text, 25% multimodal), their framework unlocked multimodal capabilities while preventing the catastrophic forgetting of foundational logic.
  • Modality Isolation (Mixture of Experts): To prevent behavioral data from entangling with semantic text processing, architectures like the one proposed by Reza Shirkavand et al. utilize dedicated expert modules and static gating. By routing non-textual modalities through specialized parameters, the model achieves high cross-modal accuracy without altering the core language weights.

Ultimately, cross-modal interference proves that building a true generalist model is not as simple as concatenating multimodal tokens into a shared transformer. Without explicit architectural boundaries or balanced optimization strategies, adding new ways to perceive the world will continue to erode the structured logic of the language models we build upon.

Key Foundational Work

Spectral Graph-Based Method of Multimodal Word Embedding (2017) — contributed 1 concept elements to this direction.
Investigating Inner Properties of Multimodal Representation and Semantic Compositionality With Brain-Based Componential Semantics (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multimodal_interference_cross_modal_inte_babf07b1,
  title = {multimodal interference (cross-modal interference)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multimodal-interference-cross-modal-interference/rdr_babf07b141c24a3c8cc7795c3c786000},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_multimodal_interference_cross_modal_inte_babf07b1_v1,
  title = {multimodal interference (cross-modal interference)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multimodal-interference-cross-modal-interference/rdr_babf07b141c24a3c8cc7795c3c786000/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake