Small vision-language models, the ones that fit on a phone or a single modest GPU, are usually trained by knowledge distillation: a large "teacher" model's next-word probabilities become soft targets for a smaller "student." These models describe images with fluent text. They also hallucinate, meaning they name objects, colors, or relations that are not in the picture. Much of this comes from the language model's prior: "kitchen" makes "refrigerator" likely whether or not one is visible. This note proposes a narrow change to how the student is taught. At the word positions where the image should decide the answer, the student is trained to reproduce the teacher's ordering of a short, chosen list of candidates. The list contains the words the image supports and the words the language prior alone would suggest. The student is not trained to copy the teacher's exact probabilities.
The underlying worry is that standard probability matching spends most of its gradient on the high-probability bulk of the distribution. That bulk is where prior-driven words sit. The decision that separates a grounded word from a plausible distractor is often a small logit gap, and that gap is exactly what a capacity-limited student can afford to lose. We want to test whether this mechanism is real, and whether rank transfer on a deliberately chosen candidate list protects the "visual tiebreak" better than temperature-tuned matching on the same list.
Core Claim
During distillation, the teacher is run a second time with the image removed. This shows which words the teacher would predict from text alone. At each position, a candidate list is formed from the teacher's top predictions with the image together with its top predictions without the image. The student is then trained with a Plackett–Luce ranking loss, a list-wise loss that models the probability of an ordering as a sequence of "pick the best remaining item" choices, to reproduce the teacher's image-conditioned order over that list. Each position is weighted by how much the image actually changed the teacher's ranking.
The claim is that this approach lowers generative hallucination (CHAIR, AMBER generative, ObjectHal-Bench) in a compact student relative to KL-distilled students at equal compute, without losing general capability. It should also show the gain specifically on prior-driven errors: adversarial and co-occurrence distractors, attributes, and relations.
Writing $s$ for student logits, $S_t$ for the candidate union at position $t$, $\pi$ for the teacher's image-conditioned ordering of $S_t$, and $\tau_t$ for Kendall's rank correlation between the teacher's with-image and without-image orderings of $S_t$, the added objective is
$$ \mathcal{L}_{\text{rank}} = -\sum_t w_t \sum_{j=1}^{|S_t|}\Big[s_{\pi_j} - \log \sum_{l \ge j} e^{s_{\pi_l}}\Big], \qquad w_t = \tfrac{1}{2}(1-\tau_t). $$
The weight $w_t$ is near zero where the image did not change the teacher's mind. Function words fall in this group, and so do positions where the teacher itself is running on prior. The weight is large where vision overturned the prior. Because $|S_t|$ is a few dozen tokens, the quadratic cost of rank statistics over a full vocabulary does not arise.
Relation to Prior Work
Three ordinal distillation methods supply the machinery, and none of them targets hallucination. Guan et al.'s Kendall's τ loss (Enhancing Logits Distillation with Plug&Play Kendall's $\\tau$ Ranking Loss) adds a smoothed rank-correlation term to KL and argues that KL gradients vanish on low-probability classes. It reports +1.83% on CIFAR-100 for vanilla KD, but over full label sets, which does not scale to 32k–150k vocabularies. PLD provides the confidence-weighted Plackett–Luce loss for classification. We reuse its form, inherit its shared-vocabulary assumption, and so pair teacher and student with the same tokenizer. TSD-KD is the closest method. It ranks student-generated top-k candidates with a teacher at high-entropy text-reasoning positions. Our list is built from the teacher's with-image and without-image predictions, so it deliberately contains the prior distractors. A student's own top-k may simply omit the grounded word, and TSD-KD's list cannot cover that case. C2KD motivates the weighting: it uses Kendall correlation between modalities to decide which samples to trust. We apply that idea per token, using the teacher's with- and without-image views.
The multimodal distillation baseline is Align-TI. It distills attention to instruction-relevant image tokens and on-policy transition probabilities using distribution matching, and its released codebase gives us a training stack. Align-TI treats POPE as one of six capability scores and does not separate prior-driven errors. HKD4VLM distills hallucination detectors, not generators, so it does not address our question.
On the hallucination side, several papers motivate the design. Summary-Guided Decoding shows that grounding pressure should fall on visual-content tokens and leave syntax free. Our image-sensitivity weight is a learned, tagger-free version of that gating. Why LVLMs Are More Prone to Hallucinations in Longer Responses and POPE document drift toward text context and acquiescence bias. The Mirage of Performance Gains constrains our evaluation. It shows that contrastive decoding's POPE gains come from a shifted "Yes" rate and vanish on CHAIR. Our method resembles contrastive decoding in that it uses an image-removed view, so the parallel matters. We do not distill a contrast score, which could import the same artifact. The image-removed view only selects candidates and weights positions, while the target ordering is the teacher's ordinary image-conditioned one. Preference methods (HDPO, TPR, SymMPO) get large reductions on full-size models with curated pairs. They are complementary rather than competing in mechanism, and we treat them as a stacking baseline.
Research Program
The work has three parts.
Diagnostic. We define a prior-inversion rate: the fraction of high-weight positions where a model ranks a teacher-identified prior distractor above the teacher's top image-conditioned token. We measure it for the teacher, an undistilled student, and a KL-distilled student on held-out COCO captions and AMBER images, then check whether it correlates with CHAIR across checkpoints. The premise of the proposal is that KL distillation raises the inversion rate relative to the teacher by more than it raises overall top-1 disagreement. If that is false, the motivation collapses, and we will report it.
Training. We train a roughly 2B student from a same-family 7B–13B teacher on LLaVA-mix-665k with ShareGPT4V captions, the data used by Align-TI. The objective is standard next-token loss plus $\mathcal{L}_{\text{rank}}$, applied both under teacher forcing and on Align-TI's on-policy rollouts. The image-free teacher pass needs no visual tokens and can be cached offline, so the overhead is about one extra cheap teacher pass per sequence. We vary the list size k (8, 16, 32), since sensitivity to truncation is a known weakness of TSD-KD. We also compare the image-sensitivity weight against part-of-speech gating.
Analysis. We break results down by AMBER dimension (existence, attribute, relation) and by response length, to see whether rank transfer delays long-response drift.
Baselines
The naive reference is the same student fine-tuned on the same data with no teacher. The standard reference is token-level KL distillation over the full vocabulary, with temperature grid-searched within the same tuning budget we give our method.
The strongest existing distillation method is Align-TI run from its released code with its own hyperparameters. It reports a 73.6% six-benchmark average for its 2B student, against 68.8% for LLaVA-1.5-7B. The file contains no isolated POPE or CHAIR number for Align-TI, so we will measure these ourselves. Our method is added on top of Align-TI rather than replacing it, using identical data, steps, and GPU-hours. "Beating it" means a lower CHAIR_s and CHAIR_i and higher AMBER generative and attribute/relation scores, while staying within one point of its six-benchmark average and keeping the POPE yes-ratio within a few points of the teacher's.
The key ablation applies KL, renormalized and temperature-tuned, on exactly our candidate lists with the same $w_t$ weights. This isolates whether the ordinal form matters or whether the gains come entirely from the choice of positions and candidates.
The stacking baseline is TPR applied to the KL-distilled student. TPR reports a reduction from 53.6% to 3.4% on ObjectHal-Bench, but for full-size LLaVA-1.5-7B, not a distilled student. We will run TPR with and without our loss upstream.
For reference points on full-size models, SHARP reports CHAIR_s going from 52.8 to 34.8 on LLaVA-1.5, and SymMPO reports 87.7 AMBER F1. We use these to place the student, not as like-for-like comparisons.
Why not just…?
Why not just raise the KL temperature? Temperature flattens the teacher distribution and restores tail gradients, which directly answers the gradient-decay argument from Guan et al.. This is the most serious threat. It is settled by the key ablation: tuned-temperature KL on the same candidate lists and weights. If that matches our method, the honest conclusion is that the contribution is where to distill (prior-contrast candidates at image-sensitive positions), not ordinality. We would rewrite the claim accordingly.
Why not just upweight visual-content tokens in ordinary KL? Summary-Guided Decoding's success with token-category gating suggests that position selection alone may do most of the work. The full-vocabulary weighted-KL baseline uses the same $w_t$ and tests this. Our expectation is that it still spends mass on the bulk distribution and leaves distractor ordering under-supervised, but this is an empirical question.
Why not distill normally and then run hallucination-targeted DPO? TPR's reported 93% drop on ObjectHal-Bench is a strong argument for post-hoc preference training. TPR also reports a curation cost of about $10, so data cost is not a decisive objection. The real question is whether the two approaches stack or substitute. If our loss adds nothing once TPR has run, its value shrinks to removing one pipeline stage. We will report that outcome as it comes out.
Why not use test-time fixes such as RITUAL or contrastive decoding on the student? These multiply inference passes, which defeats the purpose of compression, and the Mirage analysis shows contrastive gains do not survive on CHAIR. We report RITUAL on the student as a reference, with latency.
Proposed Experiments
Diagnostic check. This runs first, before any new training. Does the KL-distilled student show a higher prior-inversion rate than the teacher, and does that rate track CHAIR across checkpoints? A null result here ends the project.
Main comparison. We run the baselines above and evaluate on CHAIR (COCO), AMBER generative and discriminative with per-dimension scores, ObjectHal-Bench, MMHal-Bench, HallusionBench, and POPE across all three splits. For POPE we always report the yes-ratio, per the Mirage critique. We also run Align-TI's six capability benchmarks to catch regressions.
Decisive check on mechanism. Gains should concentrate on POPE-adversarial over POPE-random, and on AMBER attribute/relation over existence. These are the splits where the prior supplies plausible distractors. Uniform gains would suggest generic regularization rather than protection of the visual tiebreak.
Teacher-error check. On items where the teacher itself hallucinates, the low-$w_t$ weighting should keep the student from copying the error. We will test whether students exceed the teacher on those items, as TSD-KD observed on SciQ in text.
References
- Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation
- Explain in Your Own Words: Improving Reasoning via Token-Selective Dual Knowledge Distillation
- C2KD: Bridging the Modality Gap for Cross-Modal Knowledge Distillation
- Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions
- HKD4VLM: A Progressive Hybrid Knowledge Distillation Framework for Robust Multimodal Hallucination and Factuality Detection in VLMs
- Mitigating Hallucinations in LVLMs via Summary-Guided Decoding
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- Evaluating Object Hallucination in Large Vision-Language Models
- The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?
- Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization
- Systematic Reward Gap Optimization for Mitigating VLM Hallucinations
- Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization