When we build a Multimodal Large Language Model (MLLM), we typically take a highly capable, pre-trained language model and "graft" a sensory encoder—such as a vision transformer or an audio processor—onto it. Intuitively, we expect the model to fuse these streams seamlessly, using the text to guide its understanding of the pixels or soundwaves, and vice versa.
However, a deep structural conflict arises during this fusion. The language model has already spent billions of tokens learning incredibly strong statistical priors about language. It knows which words follow others, what common-sense facts look like, and how dialogue flows. When it is fine-tuned on a relatively small amount of multimodal data, a phenomenon known as modality imbalance or modality bias occurs.
Instead of performing true cross-modal integration, the model often takes a computational shortcut: it relies on its internal linguistic common sense or its own generated text tokens, effectively ignoring or overriding the sensory input. As Jize Cao et al. demonstrated in Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models, multimodal transformers consistently exhibit a significant bias toward linguistic cues, attending far more heavily to text than to visual regions during decision-making. The core logic remains anchored in language-based processing, rendering the omni-modal capabilities somewhat superficial (Fuqing Bie et al. in OmniPlay: Benchmarking Omni-Modal Models on Omni-Modal Game Playing).
The Mechanics of Textual Dominance
Why exactly does language overpower vision or audio? The root cause lies in both the scale of pre-training and the dynamics of the internal attention mechanism.
In a single-author study, Renjie Pi points out in Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization that there is a massive scale disparity between text-only pre-training and vision-language alignment datasets. This causes the model to default to linguistic statistical patterns when visual signals are weak or ambiguous.
When we look inside the transformer layers during inference, this bias manifests as an attention deficit. Xiaoyu Liang et al. analyzed the deep layers of large vision-language models in Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decoding and found that the attention mechanism heavily prioritizes textual context over visual input, leaving visual tokens with only a small fraction of the total attention weight. Similarly, Jinhe Bi et al. observed in LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering that the internal attention mechanism prioritizes textual cues and previously generated tokens over visual inputs, leading to the underutilization of visual data even when it is critical for the task.
This attention imbalance leads to several distinct failure modes:
- Object Hallucination: Because the model "reads" the prompt more attentively than it "looks" at the image, it generates text based on language priors rather than actual pixels (Yexin Liu et al. in Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly; Xianwei Zhuang et al. in VASparse: Towards Efficient Visual Hallucination Mitigation via Visual-Aware Token Sparsification).
- Hallucination Snowballing: In multi-turn conversations, models treat their own previous textual dialogue history as a more reliable source of truth than the visual features, compounding errors over time even when the image directly contradicts them (as shown by the authors of Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models).
- Exploiting Linguistic Shortcuts: In tasks like navigation, models often become "direction junkies," successfully following textual instructions while completely ignoring visual landmarks (Wanrong Zhu et al. in Diagnosing Vision-and-Language Navigation: What Really Matters).
This problem is exacerbated by certain training practices. For example, Ying Shen et al. found in MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning that fine-tuning multimodal models exclusively on text-based instructions inadvertently degrades their ability to process visual information by causing the attention mechanism to favor textual inputs. Furthermore, Tianyun Yang et al. discovered in Understanding and Mitigating Hallucination in Large Vision-Language Models via Modular Attribution and Intervention that specific attention heads fail to adapt during visual instruction tuning, maintaining their original text-prediction behavior and acting as the primary source of object hallucinations.
The Reverse Imbalance: When Vision Takes Over
Modality imbalance is not a one-way street where text always wins. The core issue is a failure to properly balance and integrate modalities, meaning that under different architectural or data conditions, the opposite imbalance can occur: a non-textual modality can completely override textual constraints.
For instance, Tianyang Han et al. identified an "instinctive bias" in The Instinctive Bias: Spurious Images lead to Illusion in MLLMs, where models prioritize immediate visual pattern matching over logical textual reasoning. When presented with a "spurious" image that is topically related but factually inconsistent with the text prompt, the model instinctively follows the visual cue, overriding the factual requirements of the text. This is mirrored in the findings of Qingxiu Dong et al. in Premise-based Multimodal Reasoning: Conditional Inference on Joint Textual and Visual Clues, where models prioritize visual plausibility over textual constraints, struggling to treat a textual premise as a mandatory filter for interpreting visual data.
A different form of reverse imbalance occurs in generative tasks involving dense spatial conditioning signals, such as generating street-view images from satellite maps. As Xianghui Ze et al. demonstrated in Controllable Satellite-to-Street-View Synthesis with Precise Pose Alignment and Zero-Shot Environmental Control that dense spatial inputs can completely overshadow sparse text prompts. Because the spatial map provides a rich, dense signal, the model's attention mechanism over-indexes on it, causing the model to ignore natural language instructions regarding environmental or stylistic attributes (like weather or lighting).
Algorithmic Interventions and Re-Balancing
To build truly robust multimodal systems, researchers have developed techniques to dynamically re-balance the influence of competing modalities during both training and inference.
Re-Balancing Contrastive Decoding
One of the most effective training-free approaches is to intervene directly in the decoding phase. Since hallucinations often occur because the model over-indexes on its own previously generated tokens, methods like Re-Balancing Contrastive Decoding (RBD) by Xiaoyu Liang et al. and Vision-enhanced Penalty Decoding by Fan Yuan et al. penalize excessive self-attention to previous text tokens while rewarding visual grounding. By subtracting or penalizing the text-only output distribution from the multimodal output distribution, these decoding strategies force the model to rely on features that are uniquely supported by the sensory input.
Modality Representation Steering
Instead of modifying decoding probabilities, another approach is to correct the internal representations directly. In LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering, Jinhe Bi et al. introduced Modality Linear Representation-Steering (MoReS). This method identifies the low-dimensional subspace where the text modality dominates and applies a steering vector to amplify the visual representations. This approach achieves state-of-the-art multimodal performance while training 500x fewer parameters than traditional fine-tuning methods like LoRA.
Targeted Distillation and Preference Optimization
To fix the imbalance at its root during alignment, researchers use targeted distillation or preference learning. Zhecan James Wang et al. developed ADVL: Adaptive Distillation for Vision-Language Tasks, which distills knowledge from a well-balanced foundation model (like CLIP) into the target architecture. This explicitly narrows the attention gap between text and vision by increasing the importance of the visual modality.
On the optimization side, Renjie Pi's Bootstrapped Preference Optimization (BPO) automatically generates negative examples that exploit the model's text-only biases, teaching the model via preference pairs to ignore its internal linguistic shortcuts and remain strictly grounded in the provided sensory evidence.