Distilling Only the Image-Driven Part of a Teacher's Predictions into Diffusion Vision-Language Models

Vision-language models that write one token at a time gradually stop looking at the image. As a caption grows, attention shifts from image patches to the text already written, and the model starts naming objects that its language habits suggest rather than objects that are present. A different family of models, discrete diffusion language models (generators that start from a fully masked sequence and fill in all positions in parallel over a few refinement passes), has been reported to keep attending to the image at every step. These models are hard to train well without a teacher, however. When all positions are predicted at once, the model tends to blend several valid outputs into an incoherent one. The standard fix is to distill from a left-to-right teacher, and here the two findings collide: the teacher that makes parallel generation coherent is the same kind of model whose late-caption predictions have lost the image. Plain distillation would teach a model that looks at the image faithfully to repeat a teacher that does not.

We propose to build distillation targets offline that keep only the image-driven part of the teacher's predictions. For every position in every training caption, we run the left-to-right teacher twice, once with the image and once without, and keep its top few candidate tokens. At positions where the image changes the teacher's prediction, we sharpen the target toward what the image adds. At positions where the image does not matter, such as function words and syntax, we keep the teacher's language-only prediction as a fluency anchor. A diffusion student trained on these compact targets pays nothing extra at inference. It is the parallel model's architecture, which does not push the image further away as the caption grows, that should let it hold onto the correction across the whole caption.

Core Claim

Truncated distribution matching restricts the distillation loss to the teacher's high-probability tokens. We claim that this restriction does two jobs in our setting. It makes offline two-condition targets small enough to store. It also supplies the plausibility constraint that keeps image-versus-no-image corrections from boosting nonsense tail tokens. We further claim that a from-scratch diffusion student keeps the resulting hallucination reduction uniformly across caption position. A left-to-right student distilled on identical targets should lose the gain in later positions, because the decay mechanism is structural and not something training data removes.

The target at caption position $t$ is

$$ q_t(v) \propto p_I(v)^{1+\alpha_t}\, p_\varnothing(v)^{-\alpha_t} \quad \text{for } v \in S_t, \qquad q_t(v)=0 \text{ otherwise}, $$

where $p_I$ and $p_\varnothing$ are the teacher's next-token distributions with and without the image, and $S_t$ is the top-$p$ ($p=0.95$) set of $p_I$ capped at $k=100$ tokens. The correction strength is $\alpha_t = \alpha\,\sigma(\mathrm{VS}_t - \tau)$, a smooth switch that turns on when $\mathrm{VS}_t$ exceeds a threshold $\tau$. Here $\mathrm{VS}_t$ is the KL divergence between the truncated $p_I$ and $p_\varnothing$ on their shared support. When $\alpha_t$ is near zero, the target reduces to the truncated teacher distribution itself. The student minimizes KL$(q_t \,\|\, p_{\text{student}})$ at masked positions.

Each ingredient comes from prior work, applied here to a new problem. Pre-training Distillation for Large Language Models: A Design Space Exploration introduced the $p=0.95$, $k=100$ truncation, reporting a 4000× storage reduction without student degradation and showing that KL outperforms MSE. That work used text-only, left-to-right models and never altered the teacher's targets. Steer Where It Matters supplies the image-sensitivity score and the evidence that visual dependence is concentrated in a minority of tokens. It computes the score online with a doubled forward pass at every decoding step. We compute it once, offline, and fold it into training targets.

Research Program

The motivating evidence is the mismatch between two lines of work. Studies of left-to-right models document the loss of image attention from several angles. OPERA finds tokens over-attending to punctuation "summary" tokens. Vista-llama and Seeing Far and Clearly trace the effect to position encodings that down-weight distant image tokens. RAR shows late layers sinking attention into background patches, and shows that ordinary fine-tuning does not repair this. Against that, A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based MLLMs reports that from-scratch diffusion models such as LLaDA-V stay "visual-hungry" across denoising steps and can restore pruned visual information. The same study finds that diffusion models converted from left-to-right weights (LaViDa-Dream) lose this property. That study does no distillation.

Understanding Knowledge Distillation in Non-autoregressive Machine Translation is both motivation and constraint. It shows parallel students need a teacher to collapse competing outputs onto one mode. It also shows that the teacher's complexity must match the student's capacity: FlowSeq scored 22.0 BLEU from a small teacher versus 18.55 from a larger one. The consequence for us is that the correction must not inflate target entropy. That is a second reason to confine it to the teacher's top-$p$ head.

The cross-modal distillation direction enters through the no-image teacher. Closing the Gap Between Text and Speech Understanding in LLMs (SALAD) distills a speech student toward a text-only model to anchor language ability and avoid forgetting, using full-vocabulary targets. We use the text-only branch in two roles. At image-insensitive positions it serves as the same kind of anchor. At image-sensitive positions it is the reference that gets divided out.

The work has three stages. First, we build the target store. The teacher is LLaVA-1.5-7B, run teacher-forced with and without the image over the LLaVA instruction and COCO caption training data. We save truncated top-$p$/$k$ entries and $\mathrm{VS}_t$ per position. As a diagnostic, we also plot $\mathrm{VS}_t$ for content words against caption position; the decay literature predicts a downward trend. Second, we train students on the store. The main arm is LLaDA-V, a from-scratch diffusion model. A control arm is LaViDa-Dream, a converted diffusion model. A second control arm is a left-to-right model of matched size. All arms use the same data, token count, optimizer steps, and tuning budget. Third, we run analyses. These include a position-binned hallucination rate, attention mass on image tokens per denoising step, and target entropy versus student coherence (repetition rate). One known mismatch is that the teacher's per-position targets are conditioned only on left context while the diffusion student sees partial context on both sides. We measure its effect by comparing against targets built from teacher-generated captions, where left context is self-consistent.

Baselines

The naive reference point is each student fine-tuned on the same captions with ordinary cross-entropy on ground-truth tokens and no teacher, which is how dMLLMs are usually adapted. The next reference is plain truncated distillation: the same $p=0.95$, $k=100$ targets from the image-conditioned teacher with $\alpha_t=0$, following Pre-training Distillation. This baseline isolates the correction. We also run sequence-level distillation on teacher argmax captions, as analyzed in Understanding KD in NAT. On a data subset where storage allows, we run full-vocabulary distillation in the style of SALAD, to check whether truncation itself costs anything.

The strongest existing competitors are inference-time corrections applied to the left-to-right teacher. On LLaVA-1.5-7B, Steer Where It Matters reports CHAIR$_s$ falling from 46.0 to 31.2 and POPE average accuracy rising from 85.09% to 89.78%, at roughly twice the per-token latency. HalluTrace reports CHAIR$_1$ falling from 14.2 to 8.1 and POPE-Adv F1 rising from 85.9% to 89.6%, with visual contrastive decoding at 10.8 CHAIR$_1$. RUDDER reports 24.4% and 23.6% relative reductions in CHAIR$_S$ and CHAIR$_i$ at 96.0% throughput. We rerun TLVS, HAD, VCD, and RUDDER with our evaluation scripts, sampling settings, and caption-length limits, since their reported numbers come from different protocols. There is no reported result on Lacuna for LLaDA-V or LaViDa-Dream on CHAIR or POPE, so those starting points are measured here.

Beating the strongest baseline has a concrete meaning: a distilled LLaDA-V that matches or improves on HAD's CHAIR$_1$ and TLVS's CHAIR$_s$ under our protocol, at single-pass inference cost, with MME and DocVQA within noise of the plain-distillation student. We treat the matched-size left-to-right student trained on the same corrected targets as a full baseline, not a side ablation, because it is the most direct threat to the claim that architecture matters. The key-ingredient ablations are $\alpha_t$ fixed (ungated correction), correction applied over the full vocabulary instead of $S_t$, and gating by a random score with the same sparsity.

Why not just…?

Why not just run contrastive decoding or TLVS on the teacher at inference? These methods work, as the numbers above show. They pay a second forward pass on every generated token, and VISTA-style steering keeps only about 58% throughput according to RUDDER. Our method pays the two-pass cost once, offline. The comparison is fair only if quality holds up, so we report hallucination and wall-clock latency together. If the distilled student is worse than TLVS at equal latency, the offline approach has failed.

Why not just distill the corrected targets into a left-to-right student? This is the real threat. If it matches the diffusion student, the architecture adds nothing and the contribution reduces to offline contrastive distillation. The decay literature (Vista-llama, RAR) predicts a specific failure: the left-to-right student's gain should concentrate in early caption positions and shrink as captions lengthen, while the diffusion student's gain should stay flat. The position-binned hallucination rate settles this. If the two curves coincide, we report that result and recommend the simpler left-to-right student.

Why not just use LLaDA-V as-is, since it already attends to the image? Attending to the image is not the same as naming the right objects. The file contains no hallucination numbers for dMLLMs. Parallel students also depend on a teacher for coherence (Understanding KD in NAT). The no-teacher and plain-distillation baselines measure both effects directly.

Why not just fine-tune on clean captions? RAR found that ordinary fine-tuning does not undo late-layer attention sinking in left-to-right models. For diffusion models the question is open, which is why ground-truth fine-tuning is our naive baseline and not an assumed failure.

Decisive Checks

The main evaluations are CHAIR$_s$/CHAIR$_i$ on COCO captioning with fixed length caps of 64, 128, and 256 tokens, POPE in its random, popular, and adversarial splits, and AMBER. MME, DocVQA, and InfoVQA guard against losing general ability. Three results decide the proposal. The first is whether the gated correction beats plain truncated distillation on the same diffusion student. If it does not, the core idea is wrong. The second is whether the diffusion student's advantage over the left-to-right student grows with caption length, the signature of the architectural claim. The third is whether LaViDa-Dream behaves like the left-to-right student and not like LLaDA-V. The redundancy study's finding that converted models keep left-to-right tendencies predicts that it will, and confirming it would tie the gain to how the model was trained from scratch, not to diffusion decoding as such. Secondary checks are that confining the correction to $S_t$ keeps repetition rates at the plain-distillation level, and that the target store stays within the storage budget reported for top-$p$/$k$ truncation.