Multimodal Large Language Models (MLLMs) are frequently celebrated for their ability to reason across text, vision, and audio. However, a persistent architectural reality complicates their evaluation: these models are built directly on top of highly capable, text-only Large Language Models (LLMs). Because their foundational language components possess immense world knowledge and strong linguistic priors, MLLMs can often "cheat" on multimodal benchmarks. They can bypass complex visual or auditory data entirely, arriving at the correct answer by exploiting statistical patterns, context clues, or logical shortcuts hidden within the text prompt itself.
To ensure that benchmarks measure genuine cross-modal comprehension rather than text-based pattern matching, researchers have developed a rigorous quality control paradigm: blind-modality filtering. By systematically evaluating whether a question can be solved in the absence of its primary sensory context, this methodology isolates true multimodal necessity.
The Illusion of Multimodal Intelligence
Consider a question from a medical or engineering benchmark accompanied by a complex diagram. If the text prompt contains highly specific terminology, a powerful language model can often deduce the correct answer using domain-specific common sense alone, completely ignoring the diagram. This vulnerability was highlighted early on by Bhavan Jasani et al. in Are we Asking the Right Questions in MovieQA?, where the authors demonstrated that video question-answering frameworks frequently failed to isolate visual reasoning from simple textual pattern matching.
When an evaluation dataset contains these linguistic shortcuts, a model's high score creates an illusion of multimodal competence. In reality, the model is simply demonstrating its text-only proficiency. To combat this, modern benchmark creation relies on filtering mechanisms that explicitly test for text-only guessability before a sample is finalized.
[Full Multimodal Prompt] ───► Separate Text from Media ───► [Text-Only Prompt]
│
▼
Evaluate with Text-Only LLM
│
┌────────────────────────────┴────────────────────────────┐
▼ ▼
[Model Answers Correctly] [Model Fails to Answer]
│ │
▼ ▼
Linguistic Shortcut Detected! True Multimodal Dependency!
(Discard or Rewrite Sample) (Keep Sample)
The Mechanism of Blind-Modality Filtering
The core execution of blind-modality filtering is straightforward: decouple the text prompt from its associated media asset (image, audio, or video), feed the isolated text into a high-capacity text-only language model, and observe the output.
If the text-only baseline successfully answers the question, the sample is compromised because the non-textual modality is mathematically redundant. Various implementations of this strategy exist across diverse domains:
- Discipline-Specific Knowledge: In expert-level benchmarks like MMVU: Measuring Expert-Level Multi-Discipline Video Understanding, Yilun Zhao et al. implemented vision-dependency filtering to remove examples where discipline-specific questions could be solved via prior textual knowledge alone. Similarly, Ming Li et al. utilized text-only baselines in EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark to discard circuit problems that were solvable purely through text descriptions.
- Document and Image Understanding: For long document analysis, Jian Chen et al. deployed multimodal-necessity filtering in VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding, forcing models to actively parse figures and layouts. In general image benchmarks, Haowei Liu et al. applied unimodal bias filtering in MIBench: Evaluating Multimodal Large Language Models over Multiple Images to guarantee that multi-image reasoning tasks genuinely required visual comparison.
- Video and Audio-Visual Integration: In long-form video understanding, Agrim Gupta incorporated blind logic filtering into HourVideo: 1-Hour Video-Language Understanding to prevent models from exploiting linguistic context over hour-long spans. For synchronized sensory inputs, Ziwei Zhou et al. used a text-only model in Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities to filter out questions that did not strictly require cross-modal temporal alignment.
By acting as a gatekeeper, this process ensures that every remaining question possesses a strict cross-modal dependency.
Advanced Isolation: Captions vs. Raw Pixels
Hiding the raw media asset is a powerful first step, but linguistic shortcuts can also manifest more subtly. For instance, if a text prompt contains a highly descriptive question, a text-only model might fail, but a text-only model provided with a standard image caption might succeed. If a caption provides enough information to solve a problem, the benchmark is merely testing text-based logic rather than raw visual reasoning or spatial imagination.
To address this deeper layer of text leakage, Yunzhuo Hao introduced modality isolation filtering in Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark. This technique operates as a two-stage filter:
- Pure Text Filtering: Discard questions answerable by a language model using only the prompt.
- Caption-Augmented Filtering: Generate detailed textual descriptions of the visual assets. If a text-only model can solve the question when paired with these captions, the sample is still removed.
This ensures that the remaining tasks strictly require the processing of raw visual structures—such as geometric configurations, spatial relationships, or pixel-level details—that defy simple textual reduction.
From Discarding to Iterative Rewriting
While discarding text-solvable questions creates a clean benchmark, it can be highly inefficient, often wasting valuable human-annotated data. A more sophisticated approach moves from passive filtering to active mitigation through iterative distractor rewriting.
In MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research, James Burgess et al. engineered an agent-based system that treats blind-modality validation as an adversarial loop. When a text-only model successfully guesses the correct answer to a multiple-choice question, the question is not thrown away. Instead, a language agent dynamically rewrites the incorrect options (the distractors).
The agent makes the new distractors linguistically identical in style, tone, and plausibility to the correct answer, effectively neutralizing any text-only statistical patterns. The modified question is then re-tested against the text-only model. This loop repeats until the text-only model's accuracy drops to random chance, ensuring that the correct answer can only be identified through precise visual analysis.
The Reality Check: What Unbiased Benchmarks Reveal
When benchmarks are rigorously scrubbed of linguistic shortcuts, the reported performance of state-of-the-art MLLMs changes dramatically.
In MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark, Xiang Yue et al. isolated and removed text-solvable questions from a premier multimodal evaluation set. Without the safety net of language priors, top-tier models experienced severe performance drops of up to 27%. This drop exposed a substantial "reasoning gap," proving that prior evaluations had significantly overestimated the models' true visual comprehension.
Similarly, when K. J. Zhu et al. introduced LIME: Less Is More for MLLM Evaluation, they utilized multimodal dependency verification to eliminate oversimplified and text-solvable examples. By reducing the dataset size by 76% and focusing exclusively on high-dependency samples, they created a more challenging, highly discriminative evaluation tool that consumes a fraction of the compute time while providing a truer reflection of multimodal capability.
Ultimately, blind-modality filtering shifts the paradigm of multimodal evaluation. By forcing models to prove they are genuinely integrating sight, sound, and text, it prevents language priors from masking visual deficiencies, paving the way for architectures that truly understand the multi-sensory world.
Key Foundational Work
Visuo-Lingustic Question Answering (VLQA) Challenge (2020) — contributed 1 concept elements to this direction.
Are we asking the right questions in MovieQA? (2019) — contributed 1 concept elements to this direction.