Eliminating Linguistic Shortcuts from Multimodal Evaluations

46 concepts43 papers1 proposal

Multimodal Large Language Models (MLLMs) are frequently celebrated for their ability to reason across text, vision, and audio. However, a persistent architectural reality complicates their evaluation: these models are built directly on top of highly capable, text-only Large Language Models (LLMs). Because their foundational language components possess immense world knowledge and strong linguistic priors, MLLMs can often "cheat" on multimodal benchmarks. They can bypass complex visual or auditory data entirely, arriving at the correct answer by exploiting statistical patterns, context clues, or logical shortcuts hidden within the text prompt itself.

To ensure that benchmarks measure genuine cross-modal comprehension rather than text-based pattern matching, researchers have developed a rigorous quality control paradigm: blind-modality filtering. By systematically evaluating whether a question can be solved in the absence of its primary sensory context, this methodology isolates true multimodal necessity.

The Illusion of Multimodal Intelligence

Consider a question from a medical or engineering benchmark accompanied by a complex diagram. If the text prompt contains highly specific terminology, a powerful language model can often deduce the correct answer using domain-specific common sense alone, completely ignoring the diagram. This vulnerability was highlighted early on by Bhavan Jasani et al. in Are we Asking the Right Questions in MovieQA?, where the authors demonstrated that video question-answering frameworks frequently failed to isolate visual reasoning from simple textual pattern matching.

When an evaluation dataset contains these linguistic shortcuts, a model's high score creates an illusion of multimodal competence. In reality, the model is simply demonstrating its text-only proficiency. To combat this, modern benchmark creation relies on filtering mechanisms that explicitly test for text-only guessability before a sample is finalized.

[Full Multimodal Prompt] ───► Separate Text from Media ───► [Text-Only Prompt]
                                                                   │
                                                                   ▼
                                                        Evaluate with Text-Only LLM
                                                                   │
                                      ┌────────────────────────────┴────────────────────────────┐
                                      ▼                                                         ▼
                         [Model Answers Correctly]                                 [Model Fails to Answer]
                                      │                                                         │
                                      ▼                                                         ▼
                         Linguistic Shortcut Detected!                             True Multimodal Dependency!
                        (Discard or Rewrite Sample)                                     (Keep Sample)

The Mechanism of Blind-Modality Filtering

The core execution of blind-modality filtering is straightforward: decouple the text prompt from its associated media asset (image, audio, or video), feed the isolated text into a high-capacity text-only language model, and observe the output.

If the text-only baseline successfully answers the question, the sample is compromised because the non-textual modality is mathematically redundant. Various implementations of this strategy exist across diverse domains:

By acting as a gatekeeper, this process ensures that every remaining question possesses a strict cross-modal dependency.

Advanced Isolation: Captions vs. Raw Pixels

Hiding the raw media asset is a powerful first step, but linguistic shortcuts can also manifest more subtly. For instance, if a text prompt contains a highly descriptive question, a text-only model might fail, but a text-only model provided with a standard image caption might succeed. If a caption provides enough information to solve a problem, the benchmark is merely testing text-based logic rather than raw visual reasoning or spatial imagination.

To address this deeper layer of text leakage, Yunzhuo Hao introduced modality isolation filtering in Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark. This technique operates as a two-stage filter:

  1. Pure Text Filtering: Discard questions answerable by a language model using only the prompt.
  2. Caption-Augmented Filtering: Generate detailed textual descriptions of the visual assets. If a text-only model can solve the question when paired with these captions, the sample is still removed.

This ensures that the remaining tasks strictly require the processing of raw visual structures—such as geometric configurations, spatial relationships, or pixel-level details—that defy simple textual reduction.

From Discarding to Iterative Rewriting

While discarding text-solvable questions creates a clean benchmark, it can be highly inefficient, often wasting valuable human-annotated data. A more sophisticated approach moves from passive filtering to active mitigation through iterative distractor rewriting.

In MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Research, James Burgess et al. engineered an agent-based system that treats blind-modality validation as an adversarial loop. When a text-only model successfully guesses the correct answer to a multiple-choice question, the question is not thrown away. Instead, a language agent dynamically rewrites the incorrect options (the distractors).

The agent makes the new distractors linguistically identical in style, tone, and plausibility to the correct answer, effectively neutralizing any text-only statistical patterns. The modified question is then re-tested against the text-only model. This loop repeats until the text-only model's accuracy drops to random chance, ensuring that the correct answer can only be identified through precise visual analysis.

The Reality Check: What Unbiased Benchmarks Reveal

When benchmarks are rigorously scrubbed of linguistic shortcuts, the reported performance of state-of-the-art MLLMs changes dramatically.

In MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark, Xiang Yue et al. isolated and removed text-solvable questions from a premier multimodal evaluation set. Without the safety net of language priors, top-tier models experienced severe performance drops of up to 27%. This drop exposed a substantial "reasoning gap," proving that prior evaluations had significantly overestimated the models' true visual comprehension.

Similarly, when K. J. Zhu et al. introduced LIME: Less Is More for MLLM Evaluation, they utilized multimodal dependency verification to eliminate oversimplified and text-solvable examples. By reducing the dataset size by 76% and focusing exclusively on high-dependency samples, they created a more challenging, highly discriminative evaluation tool that consumes a fraction of the compute time while providing a truer reflection of multimodal capability.

Ultimately, blind-modality filtering shifts the paradigm of multimodal evaluation. By forcing models to prove they are genuinely integrating sight, sound, and text, it prevents language priors from masking visual deficiencies, paving the way for architectures that truly understand the multi-sensory world.

Key Foundational Work

Visuo-Lingustic Question Answering (VLQA) Challenge (2020) — contributed 1 concept elements to this direction.
Are we asking the right questions in MovieQA? (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_text_only_baseline_filtering_a72ad6b2,
  title = {text-only baseline filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/text-only-baseline-filtering/rdr_a72ad6b2d7af45acbb29a6bcc75f2a8a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_text_only_baseline_filtering_a72ad6b2_v1,
  title = {text-only baseline filtering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/text-only-baseline-filtering/rdr_a72ad6b2d7af45acbb29a6bcc75f2a8a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake