Small vision-language models, systems that read an image and answer questions or write captions about it, are usually made by distillation: a small "student" model is trained to copy the full next-word probability distribution of a large "teacher." These small models hallucinate a lot. They describe objects that are not in the picture, typically objects that usually appear alongside what is there, such as a mouse next to a laptop. This note asks where those hallucinations come from. Part of the answer should be the teacher's soft probabilities themselves. When a teacher sees a kitchen, it puts some probability on "refrigerator" whether or not one is visible. Distillation theory treats exactly this soft mass as the useful signal ("dark knowledge"), yet for grounding it may be the channel through which the teacher's language habits pass to the student.
We propose to measure that channel directly and then close it. The measurement compares a teacher's predictions on an image with its predictions on a copy where one object has been digitally removed (inpainted). The intervention is counterfactual-contrast distillation. The student is trained to match how the teacher's predictions change when visual evidence changes, instead of only matching the teacher's predictions on the original image. Where the teacher's predictions do not change when an object is erased, the teacher is guessing from context. On those tokens the student is told not to copy it. If this works, the student gains the grounding benefit of test-time contrastive decoding at single-pass inference cost, and without inheriting the teacher's hidden hallucinations.
Core Claim
The claim has two parts. The first is diagnostic. In vision-language distillation, a large share of the student's object hallucinations are inherited: they sit on the same items where the teacher hallucinates, beyond what an independently trained student of the same size shows. The inheritance should be carried mainly by the teacher's non-top-choice probability mass on co-occurring objects, not by its argmax outputs. The second part is constructive. A distillation loss that matches the teacher's image-induced change in log-probabilities between an original and an object-removed image, and that suppresses targets on tokens the teacher does not visually ground, should cut inherited hallucination on adversarial object probing and hidden-hallucination tests. It should do this without the general-ability loss that preference optimization incurs.
This sits on an open tension in the distillation literature. A statistical perspective on distillation and Knowledge Distillation Performs Partial Variance Reduction explain the benefit of soft targets as smoother, lower-variance gradients. Knowledge Distillation \approx Label Smoothing: Fact or Fallacy? shows the benefit is instance-specific rather than uniform smoothing. These papers motivate the proposal. If the instance-specific mass encodes co-occurrence priors, the same property that reduces variance should also transmit bias, and nobody has separated the two.
Research Program
Evidence that behavior transfers. What Knowledge Gets Distilled in Knowledge Distillation? shows that distilled image classifiers inherit their teacher's attention maps, adversarial blind spots and demographic biases. In one of its tests, the fooling rate rose from 44.2% for an independent student to over 50% for a distilled one. That paper only studies classifiers, not models whose failure is a language prior overriding pixels.
Evidence that fidelity and accuracy can diverge. On the Generalization vs Fidelity Paradox in Knowledge Distillation finds that text students reach correct answers by reasoning paths different from the teacher's, and names multimodal settings as an open extension.
The closest applied work. LLaVA-MoD distills a 2B mixture-of-experts student with token-level KL followed by KTO preference distillation. It reports strong Object HalBench results but does not analyze what the student inherited or why.
The closest counterfactual work. Counterfactual Knowledge Distillation (CFKD) uses generated counterfactual images and an expert oracle to steer a classifier away from spurious features. We differ from it in three ways: our targets are autoregressive token distributions rather than class labels; the signal comes from the teacher's own sensitivity to object removal rather than a human or oracle audit; and the counterfactual images come from inpainting as in MERLIM rather than from a normalizing flow.
Stage 1: measuring inheritance. We take a teacher and student that share a tokenizer, in the Qwen-2-family setting LLaVA-MoD uses, since token-level KL requires a shared vocabulary. Students are trained under five matched regimes:
- supervised fine-tuning on ground truth, which serves as the independent student;
- sequence-level distillation on the teacher's greedy outputs;
- token-level forward KL;
- reverse KL;
- token-level KL where the teacher's non-top mass is replaced by uniform smoothing of equal entropy.
For each student we compute an inheritance score. This is the rate at which the student hallucinates on POPE and MERLIM items where the teacher hallucinates, minus the rate for the independent student, normalized by the teacher's error rate. The MERLIM paired original/inpainted images let us separate visible hallucinations from hidden ones, where a model answers "yes" correctly but keeps answering "yes" after the object is removed. The hypothesis predicts a specific ordering. Token KL should show the highest inheritance. The entropy-matched smoothing control should remove most of the excess. Sequence-level distillation should fall in between.
Stage 2: counterfactual-contrast distillation. Let $v$ be a training image, $v'$ the same image with one annotated object inpainted away, $x$ the prompt and response prefix, and $p_T$ and $p_S$ the teacher and student next-token distributions. The teacher's image effect at a position is $\Delta_T = \log p_T(\cdot \mid x, v) - \log p_T(\cdot \mid x, v')$, and $\Delta_S$ is the same quantity for the student. The loss is
$$ \mathcal{L} = \mathrm{KL}\big(\tilde p_T(\cdot \mid x, v) \,\|\, p_S(\cdot \mid x, v)\big) + \lambda \,\big\| \Delta_S - \Delta_T \big\|_2^2, $$
where $\lambda$ weights the contrast term. The contrast term makes the student respond to visual evidence at least as the teacher does. The first term uses a corrected target $\tilde p_T$ at positions where the teacher is image-insensitive for the removed object's token $o$. We define image-insensitive as $\Delta_T[o] < \tau$ for a threshold $\tau$ tuned on held-out MERLIM pairs. At those positions, $\tilde p_T$ is the teacher distribution with mass on $o$ scaled down by $\exp(\Delta_T[o]-\tau)$ and then renormalized. Everywhere else $\tilde p_T = p_T$.
The pairs come from inpainting annotated objects in the distillation images. Inference stays a single forward pass. We will also examine whether the student's middle-layer states become more separable between faithful and hallucinated outputs. SHARP found such separability peaking around layers 10–15, so this checks whether the intervention changes representations rather than only outputs.
Baselines
Naive and independent references. The naive reference is the undistilled student trained with supervised fine-tuning on the same images and ground-truth text. It is the independent student that anchors the inheritance score. The plain token-KL student is the "do nothing new" distillation reference. All students share data, steps, learning-rate sweep size and GPU hours.
Strongest existing method. This is the full LLaVA-MoD recipe, token-KL distillation followed by KTO preference distillation, run on our student at matched compute. LLaVA-MoD reports an average benchmark score of 61.6 for its 2B model versus 62.1 for LLaVA-1.5-7B. For Object HalBench it reports a win over RLHF-V but no numeric breakdown is available (no reported result found on Lacuna for the exact Object HalBench numbers). Beating it means a lower POPE-adversarial error and a lower MERLIM precision drop, with general scores on MME and SEED-Bench within noise of LLaVA-MoD's.
Inference-time decoding methods. We run visual contrastive decoding and RITUAL on the plain token-KL student. They are reported both at raw quality and at matched inference FLOPs. For reference, RITUAL lifts LLaVA-1.5 POPE accuracy from 84.13% to 88.87%. DCD reports POPE-adversarial F1 of 85.11% against 83.63% for LLaVA and about 82% for DPO. SHARP reports LLaVA-v1.5 sentence-level CHAIR falling from 52.8 to 34.8. These numbers are for 7B-scale models, not our students, so they bound what decoding can buy rather than serving as direct targets. Matching RITUAL's POPE gain on our student with one forward pass instead of two would count as beating it on the deployment-relevant axis.
Ablations. Four ablations isolate the key ingredient:
- the contrast term alone, with no target correction;
- the correction alone, with no contrast term;
- the contrast computed against a blank or noised image instead of an inpainted one;
- counterfactual pairs chosen at random rather than by co-occurrence.
The blank-image variant is what separates our idea from simply distilling visual contrastive decoding.
Why not just…?
Why not just run contrastive decoding or RITUAL on the small student? These methods need no training, but they at least double per-step cost. SHARP measures visual contrastive decoding at 13.97 s per sample versus 7.04 s for steering on LLaVA-Next. That cost lands exactly where small students are deployed. The more important point is that decoding corrects the prior at inference and cannot remove hidden hallucinations that come from context. Removing a blank image does not reveal which specific object the model is guessing. The matched-FLOPs comparison and the MERLIM hidden-hallucination breakdown settle this. If RITUAL on the student matches our MERLIM numbers at equal cost, the training-time approach is not justified.
Why not just add inpainted images with "no, it is not there" answers to supervised fine-tuning? This is the real threat. It uses the same counterfactual images, with hard labels and no teacher. Our argument is that it teaches the answer to existence questions and not grounding in free-form captions. Hard targets also discard the instance-specific signal that distillation theory credits. We run exactly this baseline, with the same pairs and compute. If it matches us on CHAIR and Object HalBench as well as on POPE, we will conclude the teacher contrast adds nothing, and the paper becomes a data-augmentation result plus the Stage 1 diagnosis.
Why not just apply DPO or KTO to the student? Preference optimization reduces hallucination. However, DCD documents a 1–4% SEED-Bench drop under DPO, and LLaVA-MoD reports DPO instability at small scale. The LLaVA-MoD-with-KTO baseline covers this alternative directly. The Stage 1 inheritance score adds something neither approach gives: whether preference tuning removes inherited errors or only lowers the overall "yes" rate. POPE showed that "yes" rates near 99% can mask this.
Why not just treat this as label smoothing? If uniform smoothing of equal entropy matched token KL on both accuracy and inheritance, the dark-knowledge channel would be empty and our correction pointless. That outcome would contradict the label-smoothing paper's conclusion in this setting. The Stage 1 smoothing control tests this before any method is built.
Proposed Experiments
Test sets and metrics. Evaluation uses:
- POPE random, popular and adversarial splits on MSCOCO, A-OKVQA and GQA, with accuracy, F1 and "yes" rate;
- MERLIM precision drop under object removal, alongside its prompt-variation spread, since MERLIM reports up to 26.5% sensitivity to prompt syntax;
- CHAIR, sentence- and instance-level, and Object HalBench for long captions;
- MME and SEED-Bench for general ability.
Decisive check 1. Stage 1 must show that token-KL inheritance exceeds both the independent student and the entropy-matched smoothing control, with confidence intervals from bootstrapping over items and three seeds. If it does not, the transmission hypothesis fails and Stage 2 is not motivated.
Decisive check 2. Counterfactual-contrast distillation must beat both the hard-label inpainting fine-tuning baseline and the blank-image contrast ablation on CHAIR and on the MERLIM hidden-hallucination drop. It must also stay within noise of LLaVA-MoD+KTO on MME and SEED-Bench.
Scaling check. Scale matters here. The fidelity-paradox paper found distillation gains shrink from up to 10.4% below 1B to about 1.3% at 7B. We therefore test students at two sizes and expect inheritance, and our improvement, to be largest for the smaller student.
References
- A statistical perspective on distillation
- Knowledge Distillation Performs Partial Variance Reduction
- Knowledge Distillation \approx Label Smoothing: Fact or Fallacy?
- What Knowledge Gets Distilled in Knowledge Distillation?
- On the Generalization vs Fidelity Paradox in Knowledge Distillation
- LLaVA-MoD: Making LLaVA Tiny via MoE-Knowledge Distillation
- Towards Fixing Clever-Hans Predictors with Counterfactual Knowledge Distillation
- Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models
- SHARP: Steering Hallucination in LVLMs via Representation Engineering
- RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in LVLMs
- Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models