Abstract
Current Multimodal Large Language Models (MLLMs) often exhibit "visual blindness," relying more on textual patterns and linguistic priors than actual pixel data. This paper explores the architectural and benchmark-related bottlenecks that prevent true visual understanding and proposes a roadmap to elevate vision to an equal footing with language.
Introduction: The Illusion of Vision
Despite the impressive reasoning capabilities of models like GPT-4V and LLaVA, a critical flaw persists: "vision" often remains a second-class citizen. These models frequently fail at basic visual tasks—such as counting intersecting lines or distinguishing "left" from "right"—that are trivial for humans. This phenomenon, termed textual over-reliance, occurs because MLLMs are built upon powerful Large Language Models (LLMs) that "hallucinate" answers based on expected textual correlations rather than visual evidence.
Categorizing Visual Understanding
To address these shortcomings, research systematically categorizes MLLM performance into four primary domains: Object & Attribute Understanding, Scene & Spatial Understanding, Higher-Level Reasoning, and Human-Centric/Temporal Understanding.
Architectural Bottlenecks
The standard MLLM pipeline typically comprises a Vision Encoder (often CLIP), an Adapter to bridge modalities, and the LLM. However, this design introduces several limitations:
- The CLIP Bottleneck: While CLIP excels at high-level semantics, it often discards fine-grained spatial details, leading to "model blindness."
- Resolution vs. Efficiency: Higher resolution improves perception but causes a "token explosion" that increases computational costs.
- Late Fusion Failures: Projecting vision into the LLM's space via a simple adapter may be fundamentally limited compared to deeper, earlier fusion.
Benchmarking and the "75% Rule"
Evaluation methods often mask these flaws. A startling finding suggests that removing the vision component of an MLLM allows it to retain up to 75% of its performance on certain benchmarks, highlighting the extent to which models "cheat" using language priors. New benchmarks like MMStar aim to ensure "visual dependency," where questions cannot be answered by the LLM using text alone.
Key Insights and Results
- Spatial Reasoning Weakness: Models often perform better on navigation when given a text description of a map rather than the image itself.
- Catastrophic Forgetting: The alignment process can degrade the standalone classification accuracy of the vision encoder.
- Mixture-of-Encoders (MoE): Using specialized encoders for different features (e.g., semantics vs. low-level details) significantly outperforms single general-purpose encoders.
Future Directions and Research Questions
The path forward involves rethinking the "plug-in" nature of vision in MLLMs. Key research questions (RQs) focus on developing unified architectures, better self-alignment for vision, and improving efficiency for real-time applications.
In conclusion, for MLLMs to achieve true multimodal intelligence, researchers must move beyond "words over pixels" and build architectures that truly perceive the visual world.
Summary updated