Quantifying Visual Reliance in Multimodal Large Language Models

This direction is split into other directions: modality ablation. Its last version stays readable and citable here.

Version 1 of 1237 papers256 conceptsfrozen 2026-10-04

When we prompt a Multimodal Large Language Model (MLLM) with an image and a question, we assume the model "sees" the image to formulate its answer. However, a growing body of evidence suggests that these models often suffer from a multimodal illusion: they frequently ignore the visual input entirely, relying instead on textual shortcuts and linguistic priors to guess the correct response.

The Multimodal Illusion

The core problem is that MLLMs are primarily built upon powerful Large Language Models (LLMs). Because these underlying models have already "read" vast amounts of text, they are incredibly adept at predicting likely answers based on the context of a question alone. Shuo Chen and colleagues demonstrated in Understanding and Improving In-Context Learning on Vision-language Models that while models appear to learn from visual examples in their prompts, they are often just using the accompanying text to identify the expected output format.

This reliance on text creates a "vision-text gap." In specialized domains like medicine, Siyuan Dai found that models sometimes perform better when images are removed, suggesting that visual data can actually act as a distractor or "noise" that confuses the model's internal linguistic reasoning (Why Text Prevails: Vision May Undermine Multimodal Medical Decision Making).

Diagnosing Reliance via Modality Ablation

To determine if a model is truly "seeing," researchers use a technique called modality ablation. By systematically removing or corrupting parts of the input, they can measure how much the model's performance depends on each stream. Common diagnostic methods include:

Why Models Cheat

Models "cheat" because textual patterns are often easier to exploit than complex visual-spatial relationships. Jiayu Wang introduced SpatialEval, a benchmark showing that even top-tier models like GPT-4o struggle with basic spatial reasoning—like navigating a grid—often performing better when the grid is described in text rather than shown as an image (Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models).

Similarly, in engineering and chemistry, models often fail to extract critical data from diagrams. Qian Wan found that in the SoM-1K benchmark, models reached higher accuracy when given expert text descriptions of engineering diagrams than when they had to interpret the diagrams themselves (SoM-1K: A Thousand-Problem Benchmark Dataset for Strength of Materials).

Toward Genuine Multimodal Integration

To move past the multimodal illusion, researchers are developing frameworks that enforce tighter grounding between pixels and text.

  1. Visual Reflection: Jian Pu introduced Reflection-V, which trains models to "look again" at specific image regions during long reasoning chains, preventing them from drifting into purely linguistic hallucinations (Look Again, Think Slowly: Enhancing Visual Reflection in Vision-Language Models).
  2. Fine-Grained Alignment: The FiVL framework, developed by Estelle Aflalo, uses automated pipelines to create segmentation datasets that link image patches directly to vocabulary tokens, forcing the model to ground its outputs in specific visual evidence (FiVL: A Framework for Improved Vision-Language Alignment).
  3. Strategic Example Selection: Even if models don't use images for reasoning, visual context is vital for retrieval. Shuo Chen developed MMICES, a strategy that uses both visual and textual similarity to select the best in-context demonstrations, significantly boosting performance by providing more relevant "blueprints" for the model to follow.

By moving beyond simple accuracy metrics and using modality-aware diagnostics, the field is shifting from building models that can see to models that must see to succeed.

Papers in this version

Showing 60 of 237 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_quantifying_visual_reliance_in_multimoda_8b555989,
  title = {Quantifying Visual Reliance in Multimodal Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quantifying-visual-reliance-in-multimodal-large-language-models/rdr_8b555989133c4c17b11ab1181e274e21},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_quantifying_visual_reliance_in_multimoda_8b555989_v1,
  title = {Quantifying Visual Reliance in Multimodal Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quantifying-visual-reliance-in-multimodal-large-language-models/rdr_8b555989133c4c17b11ab1181e274e21/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}