When we prompt a Multimodal Large Language Model (MLLM) with an image and a question, we assume the model "sees" the image to formulate its answer. However, a growing body of evidence suggests that these models often suffer from a multimodal illusion: they frequently ignore the visual input entirely, relying instead on textual shortcuts and linguistic priors to guess the correct response.
The Multimodal Illusion
The core problem is that MLLMs are primarily built upon powerful Large Language Models (LLMs). Because these underlying models have already "read" vast amounts of text, they are incredibly adept at predicting likely answers based on the context of a question alone. Shuo Chen and colleagues demonstrated in Understanding and Improving In-Context Learning on Vision-language Models that while models appear to learn from visual examples in their prompts, they are often just using the accompanying text to identify the expected output format.
This reliance on text creates a "vision-text gap." In specialized domains like medicine, Siyuan Dai found that models sometimes perform better when images are removed, suggesting that visual data can actually act as a distractor or "noise" that confuses the model's internal linguistic reasoning (Why Text Prevails: Vision May Undermine Multimodal Medical Decision Making).
Diagnosing Reliance via Modality Ablation
To determine if a model is truly "seeing," researchers use a technique called modality ablation. By systematically removing or corrupting parts of the input, they can measure how much the model's performance depends on each stream. Common diagnostic methods include:
- Null-Modality Evaluation: Replacing the image with Gaussian noise or a black square. If accuracy remains high, the model is relying on text priors (MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark).
- Incongruent Testing: Pairing a question with a random or mismatched image. Lei Li used this to show that many models perform equally well with unrelated images, indicating a total lack of visual integration (On Analyzing the Role of Image for Visual-Enhanced Relation Extraction).
- Visual Dependency Scores: Lin Chen proposed quantifying visual reliance by calculating the delta between a model's accuracy with and without visual input. This isolates the reasoning provided by the multimodal components from the LLM's internal world knowledge (Are We on the Right Way for Evaluating Large Vision-Language Models?).
Why Models Cheat
Models "cheat" because textual patterns are often easier to exploit than complex visual-spatial relationships. Jiayu Wang introduced SpatialEval, a benchmark showing that even top-tier models like GPT-4o struggle with basic spatial reasoning—like navigating a grid—often performing better when the grid is described in text rather than shown as an image (Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models).
Similarly, in engineering and chemistry, models often fail to extract critical data from diagrams. Qian Wan found that in the SoM-1K benchmark, models reached higher accuracy when given expert text descriptions of engineering diagrams than when they had to interpret the diagrams themselves (SoM-1K: A Thousand-Problem Benchmark Dataset for Strength of Materials).
Toward Genuine Multimodal Integration
To move past the multimodal illusion, researchers are developing frameworks that enforce tighter grounding between pixels and text.
- Visual Reflection: Jian Pu introduced Reflection-V, which trains models to "look again" at specific image regions during long reasoning chains, preventing them from drifting into purely linguistic hallucinations (Look Again, Think Slowly: Enhancing Visual Reflection in Vision-Language Models).
- Fine-Grained Alignment: The FiVL framework, developed by Estelle Aflalo, uses automated pipelines to create segmentation datasets that link image patches directly to vocabulary tokens, forcing the model to ground its outputs in specific visual evidence (FiVL: A Framework for Improved Vision-Language Alignment).
- Strategic Example Selection: Even if models don't use images for reasoning, visual context is vital for retrieval. Shuo Chen developed MMICES, a strategy that uses both visual and textual similarity to select the best in-context demonstrations, significantly boosting performance by providing more relevant "blueprints" for the model to follow.
By moving beyond simple accuracy metrics and using modality-aware diagnostics, the field is shifting from building models that can see to models that must see to succeed.