When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Models

Yinfeng Wang, Zhiyuan Yao, Zheren Fu, Lei Zhang, Zhendong MaoarXiv 2026 · arXiv

TL;DR

Multimodal Large Language Models (MLLMs) are highly sensitive to task-irrelevant text context, which consistently degrades their performance and systematically biases them toward negative answers. Crucially, this impact is not random noise but a highly structured affine transformation in the model's logit space. By understanding this geometric regularity, researchers can predict and even correct these context-induced errors using simple post-hoc calibration.

What Problem Does This Paper Address?

As Multimodal Large Language Models (MLLMs) are integrated into complex workflows like Retrieval-Augmented Generation (RAG) and multi-turn dialogue agents, they are frequently exposed to auxiliary text. This text—whether it's a retrieved background passage, a user instruction, or chat history—often contains information completely irrelevant to the visual task at hand.

While it is well-established that text-only LLMs are easily distracted by irrelevant context, how this "textual noise" impacts vision-language models remains poorly understood. This paper investigates whether task-irrelevant text distorts an MLLM's ability to make visually grounded judgments, exploring the underlying behavioral shifts and internal mechanisms.

Figure 1: Overview of the controlled intervention and its behavioral effects. (a) For each image-question pair, we compare a context-free input with a paired context- conditioned input, where an irrelevant context is in- serted. (b) Across multiple benchmarks, irrelevant con- text not only reduces predictive accuracy, but also shifts model decisions toward negative responses.

Methods

The authors establish a controlled intervention framework using binary visual judgment tasks (e.g., answering "Yes" or "No" to a question about an image). They compare a Context-Free Input (just the image and question) against a Context-Conditioned Input (where an irrelevant text snippet from WikiText is inserted). They test different prompt styles to see if framing the text as "possibly related" alters the effect.

To look under the hood, they analyze the decision margin ($m$), defined as the log-probability difference between the affirmative and negative tokens: $$m = \log P(\text{Yes}) - \log P(\text{No})$$

They discover a robust geometric regularity: the context-conditioned margin ($m_c$) can be closely approximated by an affine transformation of the context-free margin ($m_0$): $$m_c \approx a \cdot m_0 + b$$

The authors provide a theoretical account showing that this affine shift naturally emerges from softmax normalization within the model's attention layers. Finally, they leverage this regularity to design a post-hoc inverse-affine calibration method to mathematically "undo" the context-induced distortion.

Figure 2: Illustration of the margin-level analysis. (a) The decision margin is defined as the log-probability difference between the positive and negative answers. (b) Adding irrelevant context transforms the context- free margin m0 into the context-conditioned margin mc, which follows an approximate affine relation mc ≈ am0 + b. (c) The affine transformation shifts the effec- tive decision boundary from m0 = 0 to m0 = −b/a. This shift expands the Negative Region and compresses the Positive Region. As a result, samples in the Flip Region, i.e., 0 < m0 < −b/a, are likely to change from positive to negative predictions, inducing a systematic shift in decision preference.

Datasets

The core experiments evaluate four major vision-language benchmarks adapted into binary judgment formats:

  • POPE: Focuses on object existence verification and hallucination.
  • AMBER: Measures multiple hallucination dimensions (existence, attributes, relations).
  • MME: A broad diagnostic benchmark covering perception and cognition tasks.
  • GQA: Evaluates compositional visual reasoning.

To demonstrate that the phenomenon generalizes beyond binary setups, they also evaluate multiple-choice and realistic scenarios using MMStar, MMBench, VQAv2, A-OKVQA (for RAG), and VisDial v1.0 (for multi-turn dialogue).

Key Insights and Ideas

  • Irrelevant Text is Not Random Noise: Instead of acting as unstructured stochastic perturbations, irrelevant text applies a highly predictable, systematic geometric distortion to the model's internal confidence space.
  • The Negative Bias Phenomenon: The affine transformation shifts the effective decision boundary, expanding the "Negative Region." This causes models to flip weak positive predictions ("Yes") to negative ones ("No"), making them overly conservative.
  • Semantic Framing Amplifies the Effect: If the prompt explicitly frames the irrelevant text as potentially related (e.g., "The following caption is possibly related to the image..."), the slope $a$ drops significantly. This indicates that the model actively suppresses its visual confidence in favor of the text, even when the text is completely off-topic.
  • Attention Competition: The multiplicative attenuation (the slope $a$) is driven by token competition. Introducing new text tokens forces the attention mechanism to scale down the weights allocated to the original image and question tokens.

Key Results

  • Performance Drops: Across all tested models (LLaVA-1.5-7B, Qwen2-VL, InternVL3), irrelevant text consistently degraded accuracy. For example, Qwen2-VL-7B saw its accuracy on the MME benchmark plummet by 27.5 percentage points under the context-conditioned setting.
  • High Goodness-of-Fit: The affine model ($m_c \approx a \cdot m_0 + b$) achieved exceptionally high fit quality, with $R^2$ values frequently falling between 0.75 and 0.95 across various benchmarks.
  • Effective Calibration: Applying inverse-affine calibration successfully recovered a substantial portion of the lost performance. On the POPE benchmark with LLaVA-1.5, calibration restored accuracy from 78.0% back to 84.6% and slashed the prediction flip rate from 16.6% down to 6.1%.

Limitations and Future Work

  • Closed-Form Constraints: The primary analysis relies on binary or multiple-choice settings. A preliminary study on open-ended generation showed that irrelevant text makes descriptions shorter and more conservative, but fully modeling open-ended outputs remains an open challenge.
  • Sensitivity to Setup: The fitted affine parameters ($a$ and $b$) are sensitive to prompt structure, context length, and linguistic style, meaning they are setting-dependent descriptors rather than universal constants.
  • Simplistic Theoretical Account: The attention-based derivation provides a solid first-order approximation, but it does not establish a complete causal explanation of how cross-modal features interact in deeper layers.

Summary updated

Abstract

Multimodal large language models (MLLMs) are frequently exposed to auxiliary textual context, the impact of which on visually grounded tasks remains underexplored. In this paper, we investigate the influence of task-irrelevant context by formulating it as a controlled intervention within a binary visual judgment framework. By maintaining an invariant prompt structure while varying auxiliary inputs, we observe that irrelevant text consistently biases model predictions across diverse benchmarks. To move beyond performance metrics, we characterize this sensitivity through a decision margin defined by the log-probability difference between binary candidates. Our analysis reveals a robust geometric regularity: contextconditioned margins follow a consistent affine transformation of their context-free counterparts. This finding demonstrates that irrelevant context does not manifest as unstructured stochastic noise but as a estimable distortion of model preference. We further interpret the fitted affine parameters as metrics for visual commitment preservation and directional answer bias. These findings provide a margin-level diagnostic view of irrelevant-context effects in MLLMs and offer a basis for future studies on noisy-context robustness
BibTeX
@misc{wang2026when,
  author = {Yinfeng Wang and Zhiyuan Yao and Zheren Fu and Lei Zhang and Zhendong Mao},
  title = {{When Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language Models}},
  year = {2026},
  eprint = {2608.19208},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.19208},
}
Figure