Why Adding Images Degrades Vision Language Model Performance

18 concepts11 papers1 proposal

Intuition suggests that providing more information to a machine learning model should either improve its performance or, at worst, leave it unchanged. If a model is given a text prompt alongside a relevant image, it should theoretically use the visual context to ground its reasoning.

However, a growing body of research reveals a counterintuitive phenomenon known as cross-modal interference. Across domains ranging from geometry and robotics to medicine and game-playing, modern Vision-Language Models (VLMs) and Multimodal Large Language Models (MLLMs) frequently perform worse when given multimodal inputs compared to when they process text alone. Rather than synthesizing information across modalities, these models often experience a cognitive collision where one stream of data corrupts or distracts the other.

Visual Inputs as Distracting Noise

To understand why this degradation occurs, we must look at how models handle text redundancy. Many multimodal benchmarks contain textual shortcuts that allow models to guess the correct answer without actually looking at the accompanying image. When researchers eliminate these shortcuts, the underlying architectural weakness is exposed.

In MATHVERSE: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?, Renrui Zhang et al. demonstrate that when models are forced to rely on actual visual logic rather than text redundancy, their performance plummets. Instead of treating the image as a primary source of truth, the models treat visual tokens as noise, failing to effectively synthesize the mathematical logic across the text and visual modalities.

This failure is particularly acute in high-precision domains. For example, Zeren Zhang et al. studied geometry problem solving in Diagram Formalization Enhanced Multi-Modal Geometry Problem Solver and found that introducing visual diagrams alongside text consistently degraded model accuracy. Because current architectures struggle to natively integrate visual and textual information at a fine-grained level, the visual input acts as a pollutant. To circumvent this, the authors found it more effective to translate the diagram into a formal text-based language before feeding it to the model, bypassing the multimodal fusion process entirely.

The Spatial Grounding Bottleneck

The expectation that images provide crucial context is nowhere stronger than in spatial reasoning. Yet, when models are required to map visual space to linguistic concepts, the integration process breaks down.

In Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models, Jiayu Wang et al. show that VLMs regularly underperform their text-only counterparts on spatial tasks. The authors suggest that models are not reasoning jointly; instead, they are either failing to translate vision to text accurately or are being actively hindered by the visual modality when textual information would have sufficed.

This spatial grounding bottleneck has severe consequences for embodied agents that must turn perception into action:

  • Robotics: In ClevrSkills: Compositional Language And Visual Reasoning in Robotics, Sanjay Haresh et al. observed that robotic vision-language models perform more reliably when given text-only instructions than when given multimodal prompts containing reference images. The models fail to extract actionable spatial information from the images, creating a bottleneck that prevents them from translating visual context into physical commands.
  • Agentic Gaming: Similarly, Davide Paglieri et al. evaluated autonomous agents in BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games and found a recurring performance drop when visual data accompanied text descriptions. Rather than helping the agent plan, the redundant or complex spatial inputs overwhelm the decision-making process, causing confusion and leading to suboptimal actions.

Modality Conflict and Misaligned Weights

When a model is forced to process two streams of information, it must decide how to weigh them. Cross-modal interference often manifests as an inability to resolve conflicting or ambiguous signals between text and vision.

In ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction, Henry Peng Zou et al. demonstrate that when textual descriptions and visual cues provide mismatched or ambiguous information, multimodal models struggle to reconcile the inputs. This leads to lower accuracy than single-modality processing, as the architecture fails to resolve the conflicting inferences.

This misalignment is particularly dangerous in high-stakes fields like medicine. In WangLab at MEDIQA-M3G 2024: Multimodal Medical Answer Generation using Large Language Models, Augustin Toma et al. document a phenomenon called "modality conflict." They found that multimodal models suffer when trying to integrate noisy patient descriptions with high-confidence visual evidence (such as clinical photographs). The underlying architectures fail to weigh the signals appropriately, often allowing low-quality text to undermine and override clear visual features.

This is not a new problem, but rather a fundamental limitation of simple feature-fusion architectures. As early as the work by Parisa Kordjamshidi et al. in CLEF 2017: Multimodal Spatial Role Labeling Task Working Notes, researchers noted that simply adding visual features to linguistic models introduces noise that degrades accuracy unless the alignment between the two modalities is exceptionally precise.

The Compositional Breakdown

Why do models experience this interference even when they possess strong individual capabilities in both vision and language? The root cause lies in a failure of compositional integration.

Sibei Yang et al. formalize this in Discovering Compositional Hallucinations in LVLMs by identifying Simple Compositional Hallucination (SCHall). They show that a model can successfully identify an object visually and reason about a text-based concept perfectly when the tasks are isolated. However, when the tasks are combined into a single step, the model fails. The mere presence of the visual data actively interferes with the model's established text-reasoning pathways, breaking down its ability to execute them in tandem.

This breakdown is highly visible when models try to use visual context to resolve linguistic ambiguity or parse dense symbolic data:

Summary

Cross-modal interference reveals that modern multimodal models do not truly "see and reason" simultaneously. Instead, they often concatenate or project visual tokens into a shared space where they act as distracting noise, break spatial grounding, trigger modality conflicts, or disrupt established text-reasoning chains. Overcoming this bottleneck requires moving away from naive feature addition toward architectures that can dynamically weigh, isolate, and formalize cross-modal streams without letting them corrupt one another.

Key Foundational Work

ClevrSkills: Compositional Language And Visual Reasoning in Robotics (2024) — contributed 3 concept elements to this direction.
ImplicitAVE: An Open-Source Dataset and Multimodal LLMs Benchmark for Implicit Attribute Value Extraction (2024) — contributed 2 concept elements to this direction.
Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (2024) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_modal_interference_e44a4238,
  title = {cross-modal interference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-interference/rdr_e44a4238d48b418e9b3c6a08b8832969},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_modal_interference_e44a4238_v1,
  title = {cross-modal interference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-modal-interference/rdr_e44a4238d48b418e9b3c6a08b8832969/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake