TL;DR
This paper presents the first systematic reliability evaluation of Diffusion-based Large Vision-Language Models (dLVLMs), uncovering how their unique iterative denoising process fundamentally alters hallucination and bias patterns compared to traditional autoregressive models. While dLVLMs show competitive hallucination resistance and eliminate the notorious "yes-bias" of autoregressive models, they suffer from severe linguistic degradation, extreme vulnerability to answer-length biases, and catastrophic accuracy drops on underrepresented demographic groups.
What Problem Does This Paper Address?
Autoregressive (AR) Large Vision-Language Models dominate the multimodal landscape but suffer from well-documented flaws: they decode text sequentially from left to right, making them prone to compounding hallucinations, systematic "yes-biases" in binary queries, and preferences for longer text options.
Discrete diffusion vision-language models (dLVLMs) have emerged as a powerful alternative, generating text by iteratively unmasking and denoising tokens using bidirectional context. However, because this paradigm is so new, its reliability properties have remained completely uncharacterized. This paper tackles a critical question: Do the structural properties of diffusion-based generation mitigate or worsen the biases and hallucinations seen in AR models?
Methods
The authors conduct a comprehensive benchmarking suite, evaluating six prominent dLVLMs (including LLaDA-V, LaViDa, MMaDA, Dream-VL, and Dimple) against seven competitive AR baselines (such as LLaVA, Qwen2.5-VL, and InternVL2.5).
Instead of treating the models as black boxes, the authors analyze the internal mechanics of the diffusion process across four key dimensions:
- Object Hallucination: Probing binary yes/no object presence.
- Open-Ended Hallucination: Evaluating free-form image captioning.
- Demographic Bias: Measuring gender and race classification accuracy under different image cropping conditions.
- Selection Bias: Testing multiple-choice question answering (MCQA) using length-controlled answer options to isolate whether models rely on text length as a shortcut.
Crucially, they track two unique mechanistic signals during generation: token confidence (softmax probability) and the commit step (the specific denoising step where a token is assigned its final value).
Datasets
The evaluation leverages several established datasets tailored to specific failure modes:
- MSCOCO: Used for object hallucination via the POPE benchmark and open-ended captioning evaluation via the CHAIR metric.
- FairFace: Used to evaluate demographic disparities across seven racial groups and two genders under tight (facial features only) and loose (including background) crop conditions.
- CUB-200-2011 (Birds) & Stanford Dogs: Used to construct fine-grained, length-controlled multiple-choice questions to evaluate selection and length biases.
Key Insights and Ideas
- The "Commit Step" as a Hallucination Smoke Detector: In dLVLMs, tokens that are finalized late in the denoising process with low confidence are highly correlated with hallucinated objects. As shown in Figure 1, visually grounded objects (like "lantern") are committed early, while hallucinations (like "clock" and "vase") cluster at the very final denoising steps.
- Linguistic Quality vs. Grounding are Decoupled: In AR models, generation fluency and factual accuracy are tightly bound. In dLVLMs, they are separable. Forcing a diffusion model to decode in an AR-style (left-to-right) drastically improves its linguistic fluency but leaves its hallucination rate virtually unchanged.
- The Step-0 Length Prior: The extreme preference of dLVLMs for longer multiple-choice options is a "one-shot" bias. The model overwhelmingly selects the longer option at the very first denoising step (Step 0), and the subsequent iterative denoising steps almost never correct this initial blind spot.
- Reversing the Yes-Bias: AR models are notoriously agreeable, frequently answering "yes" to binary questions regardless of the image content. dLVLMs flip this behavior entirely, exhibiting a conservative "no-bias" due to their bidirectional training dynamics.
Key Results
- Object Hallucination (POPE): dLVLMs achieve accuracy competitive with modern AR models (e.g., Dimple hits 88% on Random). However, their
Yes%drops to 35–45%, contrasting sharply with older AR models that approach 100%Yes%(see Figure 4).
- Open-Ended Hallucination (CHAIR): Dream-VL achieves the lowest overall object-level hallucination rate ($CHAIR_I$: 11.69%), outperforming all AR baselines (see Figure 5). However, reducing the number of denoising steps to speed up generation causes catastrophic linguistic failures (e.g., grammatical errors jumping from 1% to 60%).
- Demographic Collapse: dLVLMs show severe fragility on underrepresented demographic groups. For instance, MMaDA-MixCoT drops to exactly 0% accuracy on Latino Hispanic and Southeast Asian faces when tightly cropped, confidently misclassifying them into dominant categories like White or East Asian.
- Extreme Length Bias: When the correct multiple-choice answer is shorter than the incorrect distractors, dLVLM accuracy collapses dramatically (e.g., LaViDa-Dream drops to 2.80% accuracy). Conversely, when the correct answer is longer, accuracy skyrockets to 90-99%.
Limitations and Future Work
The authors note several limitations in their study:
- Evaluation Scope: The analysis is restricted to four reliability dimensions, leaving out other critical behaviors like toxicity, sycophancy, and temporal reasoning.
- LLM-as-a-Judge Dependency: The linguistic quality evaluation relies on GPT-4o-mini, which has not been fully validated against human annotations for this specific task.
- Confounding Variables: Disentangling the exact impact of the generative architecture from the underlying vision encoders and training data scales will require more extensive, computationally heavy ablations in future work.
Summary updated