Words Over Pixels? Rethinking Vision in Multimodal Large Language Models

Anubhooti Jain, Mayank Vatsa, Richa Singh

Abstract

Current Multimodal Large Language Models (MLLMs) often exhibit "visual blindness," relying more on textual patterns and linguistic priors than actual pixel data. This paper explores the architectural and benchmark-related bottlenecks that prevent true visual understanding and proposes a roadmap to elevate vision to an equal footing with language.

Introduction: The Illusion of Vision

Despite the impressive reasoning capabilities of models like GPT-4V and LLaVA, a critical flaw persists: "vision" often remains a second-class citizen. These models frequently fail at basic visual tasks—such as counting intersecting lines or distinguishing "left" from "right"—that are trivial for humans. This phenomenon, termed textual over-reliance, occurs because MLLMs are built upon powerful Large Language Models (LLMs) that "hallucinate" answers based on expected textual correlations rather than visual evidence.

Categorizing Visual Understanding

To address these shortcomings, research systematically categorizes MLLM performance into four primary domains: Object & Attribute Understanding, Scene & Spatial Understanding, Higher-Level Reasoning, and Human-Centric/Temporal Understanding.

Taxonomy of visual understanding tasks in MLLMs, ranging from basic object detection to complex higher-level reasoning.

Architectural Bottlenecks

The standard MLLM pipeline typically comprises a Vision Encoder (often CLIP), an Adapter to bridge modalities, and the LLM. However, this design introduces several limitations:

  • The CLIP Bottleneck: While CLIP excels at high-level semantics, it often discards fine-grained spatial details, leading to "model blindness."
  • Resolution vs. Efficiency: Higher resolution improves perception but causes a "token explosion" that increases computational costs.
  • Late Fusion Failures: Projecting vision into the LLM's space via a simple adapter may be fundamentally limited compared to deeper, earlier fusion.

Benchmarking and the "75% Rule"

Evaluation methods often mask these flaws. A startling finding suggests that removing the vision component of an MLLM allows it to retain up to 75% of its performance on certain benchmarks, highlighting the extent to which models "cheat" using language priors. New benchmarks like MMStar aim to ensure "visual dependency," where questions cannot be answered by the LLM using text alone.

Key Insights and Results

  • Spatial Reasoning Weakness: Models often perform better on navigation when given a text description of a map rather than the image itself.
  • Catastrophic Forgetting: The alignment process can degrade the standalone classification accuracy of the vision encoder.
  • Mixture-of-Encoders (MoE): Using specialized encoders for different features (e.g., semantics vs. low-level details) significantly outperforms single general-purpose encoders.

Future Directions and Research Questions

The path forward involves rethinking the "plug-in" nature of vision in MLLMs. Key research questions (RQs) focus on developing unified architectures, better self-alignment for vision, and improving efficiency for real-time applications.

A framework of open research questions (RQs) mapped to the MLLM architecture, highlighting areas for future improvement in alignment, robustness, and efficiency.

In conclusion, for MLLMs to achieve true multimodal intelligence, researchers must move beyond "words over pixels" and build architectures that truly perceive the visual world.

Summary updated

Abstract

Multimodal Large Language Models (MLLMs) promise seamless integration of vision and language understanding. However, despite their strong performance, recent studies reveal that MLLMs often fail to effectively utilize visual information, frequently relying on textual cues instead. This survey provides a comprehensive analysis of the vision component in MLLMs, covering both application-level and architectural aspects. We investigate critical challenges such as weak spatial reasoning, poor fine-grained visual perception, and suboptimal fusion of visual and textual modalities. Additionally, we explore limitations in current vision encoders, benchmark inconsistencies, and their implications for downstream tasks. By synthesizing recent advancements, we highlight key research opportunities to enhance visual understanding, improve cross-modal alignment, and develop more robust and efficient MLLMs. Our observations emphasize the urgent need to elevate vision to an equal footing with language, paving the path for more reliable and perceptually aware multimodal models.
BibTeX
@inproceedings{jain2025words,
  author = {Anubhooti Jain and Mayank Vatsa and Richa Singh},
  title = {{Words Over Pixels? Rethinking Vision in Multimodal Large Language Models}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/1164},
}
Figure