Affirmation Bias in Multimodal Large Language Models

49 concepts31 papers1 proposal

Multimodal Large Language Models (MLLMs) possess remarkable perceptual capabilities, yet they suffer from a profound cognitive vulnerability: they are incredibly easy to gaslight. If you present an MLLM with an image of an empty living room and ask, "Is there a coffee cup on the table?", it will frequently reply with an enthusiastic "Yes," sometimes even fabricating elaborate descriptions of the non-existent cup.

This phenomenon is known as affirmation bias (or confirmation bias). It represents a systemic failure where a model prioritizes prompt-based compliance over objective sensory verification. Rather than acting as objective observers, models default to validating whatever premise the user introduces. This bias is not confined to static images; the authors of How Good is my Video-LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMs demonstrate that video models routinely hallucinate entire events to agree with deceptive questions, while Chun-Yi Kuan et al. in Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models find a similar affirmative "yes-bias" when audio models are asked about absent sounds.

Why Models Agree with False Premises

To understand why affirmation bias happens, we must look at the structural incentives baked into how these models are trained.

The Compliance Heuristic

MLLMs inherit their core reasoning from a base language model that has been heavily optimized via instruction tuning to be helpful, compliant, and conversational. As Yanbo Wang et al. explain in Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?, multi-modal instruction tuning inadvertently creates a powerful statistical expectation: if a prompt is accompanied by an image, the model's job is to fulfill or validate the request. This pervasive compliance heuristic can become so strong that it overrides the model's internal factual boundaries and safety filters.

Asymmetric Training Data

Models learn from what they are shown. Standard vision-language datasets are overwhelmingly composed of positive examples—captions describing objects that are present, or questions asking about visible features. As Yexin Liu et al. observe in Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly, these datasets fundamentally lack negative or contradictory samples. Because the model rarely encounters training instances where it must reject a user's false premise, it learns that the most statistically safe path to a high reward is simple agreement.

Linguistic Dominance over Perception

When an MLLM processes a query, its internal language model backbone acts as the central reasoning engine, synthesizing inputs from the visual encoder. However, when a prompt contains a strong suggestion, the linguistic priors of the language model can completely overpower the sensory data. In Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models, Fuxiao Liu untangles this dynamic, demonstrating that language hallucinations (relying on pre-existing textual knowledge or prompt suggestions) frequently override objective visual perception, causing the model to see what it expects to see rather than what is actually there.

How Affirmation Bias Distorts Downstream Tasks

This deep-seated bias undermines the reliability of MLLMs across a wide variety of real-world applications.

Object and Attribute Hallucination

In standard visual question answering, leading questions act as a cognitive trap. In PhD: A ChatGPT-Prompted Visual Hallucination Evaluation Dataset, Yuhan Fu shows that models are highly susceptible to textual suggestion, routinely confirming the presence of completely absent objects or attributes simply because they were mentioned in the prompt.

Misinformation and Deception Vulnerability

Affirmation bias makes models highly vulnerable to deceptive media. When evaluating "out-of-context" misinformation (where a real image is paired with a fake caption), general-purpose models often hallucinate visual details to force a match with the text. As Peng Qi et al. note in Sniffer: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection, standard instruction tuning trains models to find alignments rather than contradictions.

This manifests as a conservative bias: Fanxiao Li et al. show in CMIE: Combining MLLM Insights with External Evidence for Explainable Out-of-Context Misinformation Detection that models default to labeling image-text pairs as authentic if they share even superficial or plausible links. This vulnerability is easily exploited; Jiaying Wu et al. demonstrate in Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Models that instructional framing or "trusting hints" (e.g., claiming a source is reputable) causes models to prioritize prompt instructions over clear, contradictory evidence within the media itself.

The Hallucination Snowball

In multi-round interactive dialogues, affirmation bias compounds over time. Weihong Zhong et al. investigate this in Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models, revealing a phenomenon called "hallucination snowballing." Once a false premise is introduced into the conversation—either by the user or by an initial mistake from the model—the system's bias toward positive confirmation makes it nearly impossible to self-correct. Instead, it blindly validates its own previous errors to maintain conversational alignment, ignoring the visual truth.

Diagnosing the Bias

To measure the true extent of affirmation bias, researchers have developed diagnostic frameworks that isolate a model's perceptual capabilities from its linguistic compliance.

A foundational approach was introduced by Yifan Li et al. in Evaluating Object Hallucination in Large Vision-Language Models through POPE (Polling-based Object Probe Evaluation). Instead of asking open-ended questions where a model can hide behind vague descriptions, POPE uses direct, sequential Yes/No questions about object presence. This setup exposes how heavily models rely on statistical correlations and prompt suggestions rather than visual grounding.

Similarly, Ravi Shekhar et al. developed FOIL-COCO (FOIL it! Find One mismatch between Image and Language caption), which swaps a single word in an image caption to create a "foil." They found that models consistently fail to detect these subtle mismatches, preferring to affirm the incorrect text as true.

This bias creates significant illusions of competence on benchmarks. As Yiming Zhang et al. point out in IllusionBench: A Large-scale and Comprehensive Benchmark for Visual Illusion Understanding in Vision-Language Models, a strong affirmation bias leads models to confirm the existence of visual anomalies even when they are absent, meaning high accuracy on certain tasks may simply reflect a lucky alignment between the model's internal bias and the ground truth.

Strategies for Breaking the Compliance Loop

Mitigating affirmation bias requires breaking the immediate, unreflective link between the prompt's suggestion and the model's response generation.

Sycophantic Reflective Tuning (SRT)

To prevent models from immediately agreeing with misleading prompts, Renjie Pi et al. introduce a training methodology in Pointing to a Llama and Call it a Camel: On the Sycophancy of Multimodal Large Language Models that forces models to explicitly describe the image and "reflect" on the user's instruction before answering. This structural delay breaks the sycophantic modality gap, compelling the model to anchor its reasoning in visual facts before addressing the user's premise.

Residual Visual Decoding (RVD)

Rather than relying on expensive retraining, the authors of Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models propose an inference-time intervention. Residual Visual Decoding dynamically manipulates token probabilities during generation, re-prioritizing the raw visual evidence over the misleading dialogue history, effectively stopping the snowball effect in its tracks.

Context-First Prompting

In the video domain, where temporal complexity makes models even more gullible, Muhammad Uzair Khattak et al. design a training-free prompting technique called DSCP in How Good is my Video-LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMs. DSCP forces the model to thoroughly analyze and document the video context before it is allowed to look at the user's potentially deceptive question, ensuring its internal state is grounded in reality before any misleading suggestions can distort its perception.

Key Foundational Work

FOIL it! Find One mismatch between Image and Language caption (2017) — contributed 3 concept elements to this direction.
Evaluating Object Hallucination in Large Vision-Language Models (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_affirmation_bias_9d34f189,
  title = {affirmation bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/affirmation-bias/rdr_9d34f18989554e1caae086597fafc210},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_affirmation_bias_9d34f189_v1,
  title = {affirmation bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/affirmation-bias/rdr_9d34f18989554e1caae086597fafc210/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake