Reliability Challenges in Diffusion Vision-Language Models

Md. Atabuzzaman, Chris ThomasarXiv 2026 · arXiv

TL;DR

This paper presents the first systematic reliability evaluation of Diffusion-based Large Vision-Language Models (dLVLMs), uncovering how their unique iterative denoising process fundamentally alters hallucination and bias patterns compared to traditional autoregressive models. While dLVLMs show competitive hallucination resistance and eliminate the notorious "yes-bias" of autoregressive models, they suffer from severe linguistic degradation, extreme vulnerability to answer-length biases, and catastrophic accuracy drops on underrepresented demographic groups.

What Problem Does This Paper Address?

Autoregressive (AR) Large Vision-Language Models dominate the multimodal landscape but suffer from well-documented flaws: they decode text sequentially from left to right, making them prone to compounding hallucinations, systematic "yes-biases" in binary queries, and preferences for longer text options.

Discrete diffusion vision-language models (dLVLMs) have emerged as a powerful alternative, generating text by iteratively unmasking and denoising tokens using bidirectional context. However, because this paradigm is so new, its reliability properties have remained completely uncharacterized. This paper tackles a critical question: Do the structural properties of diffusion-based generation mitigate or worsen the biases and hallucinations seen in AR models?

Methods

The authors conduct a comprehensive benchmarking suite, evaluating six prominent dLVLMs (including LLaDA-V, LaViDa, MMaDA, Dream-VL, and Dimple) against seven competitive AR baselines (such as LLaVA, Qwen2.5-VL, and InternVL2.5).

Instead of treating the models as black boxes, the authors analyze the internal mechanics of the diffusion process across four key dimensions:

  1. Object Hallucination: Probing binary yes/no object presence.
  2. Open-Ended Hallucination: Evaluating free-form image captioning.
  3. Demographic Bias: Measuring gender and race classification accuracy under different image cropping conditions.
  4. Selection Bias: Testing multiple-choice question answering (MCQA) using length-controlled answer options to isolate whether models rely on text length as a shortcut.

Crucially, they track two unique mechanistic signals during generation: token confidence (softmax probability) and the commit step (the specific denoising step where a token is assigned its final value).

Datasets

The evaluation leverages several established datasets tailored to specific failure modes:

  • MSCOCO: Used for object hallucination via the POPE benchmark and open-ended captioning evaluation via the CHAIR metric.
  • FairFace: Used to evaluate demographic disparities across seven racial groups and two genders under tight (facial features only) and loose (including background) crop conditions.
  • CUB-200-2011 (Birds) & Stanford Dogs: Used to construct fine-grained, length-controlled multiple-choice questions to evaluate selection and length biases.

Key Insights and Ideas

  • The "Commit Step" as a Hallucination Smoke Detector: In dLVLMs, tokens that are finalized late in the denoising process with low confidence are highly correlated with hallucinated objects. As shown in Figure 1, visually grounded objects (like "lantern") are committed early, while hallucinations (like "clock" and "vase") cluster at the very final denoising steps.
Qualitative hallucination analysis showing late commit steps for hallucinated tokens.
  • Linguistic Quality vs. Grounding are Decoupled: In AR models, generation fluency and factual accuracy are tightly bound. In dLVLMs, they are separable. Forcing a diffusion model to decode in an AR-style (left-to-right) drastically improves its linguistic fluency but leaves its hallucination rate virtually unchanged.
  • The Step-0 Length Prior: The extreme preference of dLVLMs for longer multiple-choice options is a "one-shot" bias. The model overwhelmingly selects the longer option at the very first denoising step (Step 0), and the subsequent iterative denoising steps almost never correct this initial blind spot.
  • Reversing the Yes-Bias: AR models are notoriously agreeable, frequently answering "yes" to binary questions regardless of the image content. dLVLMs flip this behavior entirely, exhibiting a conservative "no-bias" due to their bidirectional training dynamics.

Key Results

  • Object Hallucination (POPE): dLVLMs achieve accuracy competitive with modern AR models (e.g., Dimple hits 88% on Random). However, their Yes% drops to 35–45%, contrasting sharply with older AR models that approach 100% Yes% (see Figure 4).
POPE benchmark evaluation table.
  • Open-Ended Hallucination (CHAIR): Dream-VL achieves the lowest overall object-level hallucination rate ($CHAIR_I$: 11.69%), outperforming all AR baselines (see Figure 5). However, reducing the number of denoising steps to speed up generation causes catastrophic linguistic failures (e.g., grammatical errors jumping from 1% to 60%).
CHAIR hallucination evaluation table.
  • Demographic Collapse: dLVLMs show severe fragility on underrepresented demographic groups. For instance, MMaDA-MixCoT drops to exactly 0% accuracy on Latino Hispanic and Southeast Asian faces when tightly cropped, confidently misclassifying them into dominant categories like White or East Asian.
  • Extreme Length Bias: When the correct multiple-choice answer is shorter than the incorrect distractors, dLVLM accuracy collapses dramatically (e.g., LaViDa-Dream drops to 2.80% accuracy). Conversely, when the correct answer is longer, accuracy skyrockets to 90-99%.

Limitations and Future Work

The authors note several limitations in their study:

  1. Evaluation Scope: The analysis is restricted to four reliability dimensions, leaving out other critical behaviors like toxicity, sycophancy, and temporal reasoning.
  2. LLM-as-a-Judge Dependency: The linguistic quality evaluation relies on GPT-4o-mini, which has not been fully validated against human annotations for this specific task.
  3. Confounding Variables: Disentangling the exact impact of the generative architecture from the underlying vision encoders and training data scales will require more extensive, computationally heavy ablations in future work.

Summary updated

Abstract

Diffusion-based Large Vision-Language Models (dLVLMs) have recently emerged as a compelling alternative to autoregressive (AR) LVLMs, offering advantages in parallel decoding, bidirectional context, and controllable generation. Despite rapid progress, their reliability properties remain largely uncharacterized. We present the first systematic reliability evaluation of hallucination and bias in dLVLMs, benchmarking six diffusion models against competitive AR baselines across four dimensions. Our key findings are: (1) dLVLMs reverse the yes-bias of AR models in binary visual queries; (2) they achieve competitive hallucination rates yet exhibit degraded linguistic quality; (3) they collapse to near-zero accuracy on underrepresented racial groups with opposite-polarity gender bias; and (4) they exhibit accuracy collapse in multiple-choice settings when the correct option is shorter than its distractors, associated with a length prior that emerges at the first denoising step. Tokens committed at late denoising steps with low confidence further correlate with hallucinated content, pointing to a mechanistic signal unique to diffusion generation. These patterns vary across model families, suggesting reliability is shaped by the generative paradigm together with training data.
BibTeX
@misc{atabuzzaman2026reliability,
  author = {Md. Atabuzzaman and Chris Thomas},
  title = {{Reliability Challenges in Diffusion Vision-Language Models}},
  year = {2026},
  eprint = {2609.01318},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.01318},
}
Figure