Small vision-language models are increasingly trained on captions written by a larger teacher model rather than by people. This proposal argues that these "student" models inherit a specific failure from their teacher's text. They learn, as memorized text patterns, which objects tend to be mentioned together: a table usually comes with chairs, a fork with a knife. Because a small model leans on its text stream far more than on its image stream, it then names those objects whether or not they appear in the picture. If this is right, removing the teacher's own mistakes from the training captions will not fix the student, because even perfectly accurate captions carry the co-occurrence pattern. The fix has to change the training data so that object words cannot be guessed from the surrounding text alone.
Concretely, we propose counterfactual-pair distillation. For a fraction of training images, an object that strongly co-occurs with the rest of the scene is erased by inpainting, the teacher re-captions the edited image, and the student trains on both versions. The same textual context now leads to different object mentions, so the only way to predict them is to look. Alongside the method, we introduce a simple diagnostic: compare how likely the student finds an object word with the real image versus a blank one. This measures the student's reliance on memorized text priors one token at a time, and lets us test whether hallucinations are exactly the tokens the student would have produced blind.
Core Claim
We make three claims, each testable on its own. First, when a compact student is distilled from teacher captions, its object hallucinations concentrate on words with high text-only probability, and that probability tracks how often the word co-occurred with the context in the distillation corpus. In other words, hallucination here is a form of memorization of caption statistics. Second, distillation makes this worse than in the teacher, and more so as the student shrinks. Third, pairing captions with counterfactually edited images reduces hallucination more than cleaning or rebalancing the corpus does, while keeping caption length and recall intact.
To make the first claim measurable, define for an object word $w$ at position $t$ the grounding margin
$$ r(w) = \log p_\theta(w \mid v, y_{<t}) - \log p_\theta(w \mid \varnothing, y_{<t}), $$
where $p_\theta$ is the student, $v$ is the image, $\varnothing$ is a blank image, and $y_{<t}$ is the preceding caption text. A margin near zero means the student would have said $w$ without looking. This is an output-level relative of the attention ratio used in Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs. That paper defines a Visual-to-Textual Attention Contribution Ratio and shows hallucinations cluster where textual attention dominates. We use the margin as a training-time diagnostic for students rather than as a decoding intervention on a frozen model.
Where This Sits
Several papers motivate the gap. Membership and Memorization in LLM Knowledge Distillation shows that text-only students inherit their teachers' membership leakage and verbatim strings. Its numbers are a membership inference attack area under the curve (AUC) of up to 0.83 under ImitKD, and 11.35% of teacher-memorized samples reappearing under SeqKD. It never asks whether inherited text memorization surfaces as hallucination in a multimodal student, which is our question.
Beyond Modality Collapse finds that, when an image-text dataset is compressed into a small synthetic set, text gradients dominate and image features collapse. It measures only retrieval, not generative hallucination. Quantifying Cross-Modality Memorization in Vision-Language Models shows that memorization is asymmetric across modalities and that unlearning fails outside the modality where a fact was acquired. That second result constrains our design: if priors are memorized through text, only an intervention that changes what text predicts should remove them, which is why we edit image–caption pairs rather than add text-side penalties.
Evaluating Object Hallucination in Large Vision-Language Models and Analyzing and Mitigating Object Hallucination in Large Vision-Language Models tie hallucination to co-occurrence statistics in the training data. They diagnose the problem or patch outputs after the fact; neither changes the training distribution to break the co-occurrence.
One paper appears to cut the other way. Hallucination is a Consequence of Space-Optimality proves that a capacity-limited model should, at the optimum, assign high confidence to some false claims. Read naively, this says small students must hallucinate and no data change can help. We take it as a constraint instead. Its argument covers memorizing arbitrary facts with no available evidence. Our setting differs because the evidence is present in the image. Counterfactual pairs make the prior-based guess costly in the loss, so the cheap strategy the theorem predicts stops being optimal. The size sweep below tests whether this holds or whether the capacity wall reasserts itself in small students.
The method borrows from Mitigating Object Hallucination via Data Augmented Contrastive Tuning. HALVA perturbs caption text to create contrastive negatives and ties the model to a frozen reference with a token-wise KL term. We keep the KL tether but perturb the image instead of the text, because text-side negatives leave the text-only prediction of an object word unchanged.
Research Program
The pipeline starts from MS-COCO training images, keeping the validation images used by CHAIR and POPE untouched. A large vision-language teacher writes detailed captions for every training image. Students of roughly 0.5B, 1.5B and 3B language parameters share one frozen vision encoder and are trained on these captions. This forms the naive distilled family.
For the counterfactual set, we use COCO instance masks to pick, per image, the present object with the highest corpus co-occurrence with the other present objects. We inpaint it out, have the teacher re-caption, and keep the pair only if an open-vocabulary detector and the caption both confirm the object is gone.
Inpainting leaves visible artifacts, and a student could learn that "an artifact means the object is absent." To block this shortcut, an equal number of images get inpainting edits over background regions with no object removed. The total number of training examples is matched across all conditions.
The analysis has two parts. The first, which needs no new method, computes $r(w)$ for every object mention the naive students generate on COCO validation. It relates the margin to three quantities: whether the mention is a hallucination, how often the object co-occurs with the context in the distillation corpus, and n-gram overlap with the distillation captions.
The memorization side follows the protocol of Membership and Memorization in LLM Knowledge Distillation, adapted to captions. We run membership inference with held-in versus held-out teacher captions. We also measure a verbatim continuation rate: given the image and a caption prefix, how often the student reproduces the rest of the caption exactly. Because that paper finds membership and verbatim memorization can disagree, we report both.
Baselines
The naive reference point is a student distilled on raw teacher captions with standard cross-entropy. Every other condition must beat it on hallucination without losing caption recall.
The strongest simple competitor is cleaned-corpus distillation. Here we delete from teacher captions every object word not supported by COCO ground-truth annotations, using the gold labels rather than a detector so the cleaning is as good as it can be. This tests the premise directly. If the cleaned student matches counterfactual-pair distillation, the proposal reduces to "filter the teacher," and we should say so.
A second data-side competitor is co-occurrence-rebalanced distillation. It reweights real training images so that co-occurring object pairs appear less correlated, using the same added-example budget as our method.
A third is modality dropout, following Wolf et al., with the dropout rate tuned on a held-out split. It is the standard training-time remedy for text dominance.
A fourth is HALVA's phrase-level contrastive tuning applied to the naive student with its released recipe Mitigating Object Hallucination via Data Augmented Contrastive Tuning. The file reports AMBER F1 gains of up to 13.4% for it on larger models; no reported result was found for it on sub-3B distilled students.
The strongest existing method overall is Owl's dual-path contrastive decoding applied at inference to the naive student. On LLaVA-1.5 it cuts sentence-level CHAIR from 47.4% to 26.2% Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs. The LURE revisor is a second post-hoc baseline; it reports MiniGPT-4 sentence-level CHAIR dropping from 26.8 to 19.7 Analyzing and Mitigating Object Hallucination in Large Vision-Language Models. Since neither number is for a compact distilled student, we rerun both on our students. Their decoding hyperparameters get the same validation tuning budget as our training hyperparameters.
Two ablations isolate the key ingredient. Removing a random low-co-occurrence object instead of the highest-co-occurrence one tests whether targeting memorized priors matters. Adding equally many unedited teacher-captioned images tests whether gains come from data volume alone.
Beating Owl means the following: a single-pass student trained with counterfactual pairs at least matches the Owl-decoded naive student on adversarial POPE F1 and sentence-level CHAIR, while keeping CHAIR-measured object recall and caption length within the naive student's range.
Why not just…?
Why not just clean the teacher's captions? This is the most serious threat, and the experiment is built around it. Our hypothesis says even fully correct captions teach "table implies chairs." The cleaned-corpus student should therefore keep near-zero grounding margins and fail on POPE's adversarial split, which samples exactly the co-occurring negatives. If the cleaned student closes most of the gap, the memorization-of-priors story is wrong and we would report filtering as the answer.
Why not just rebalance real images to flatten co-occurrence? Rebalancing can only reweight scenes that already exist. Strongly correlated pairs rarely appear apart, so there is little to upweight. Counterfactual editing manufactures exactly the missing "table without chairs" cases. The rebalanced baseline at matched budget, and the random-object ablation, show whether this difference matters in practice.
Why not just run Owl or a revisor at inference? These work. They need two forward paths per decoding step, or a second model, which defeats the purpose of a compact student. Owl also reports that naive suppression produces "shy" models with shortened outputs. We test both a match by our single-pass student and the combination of the two methods. If Owl on the naive student clearly wins, the honest conclusion is that training-time fixes are not worth it at this scale.
Why not just use a bigger student? The rate-distortion result Hallucination is a Consequence of Space-Optimality predicts slowly diminishing returns from capacity. Quantifying Cross-Modality Memorization in Vision-Language Models found that scaling Gemma-3 did not close the cross-modal transfer gap. The size sweep checks whether the hallucination gap between student and teacher shrinks on its own.
Proposed Experiments
Evaluation uses CHAIR at sentence and instance level on COCO validation, together with object recall and mean caption length to catch shy models. It also uses POPE's random, popular and adversarial splits, with F1 and the "yes" rate, given the near-99% yes bias reported in Evaluating Object Hallucination in Large Vision-Language Models. Two further benchmarks complete the set: AMBER for attribute hallucination, and ROPE's adversarial AAAAB teacher-forcing setting from Multi-Object Hallucination in Vision Language Models. In that setting, visual attention falls below 20% and accuracy on the final object drops to near zero, which is the momentum version of our prior-reliance story. The reference points for orientation are MiniGPT-4's POPE F1 of 78.86 on the random split and 71.37 on the adversarial split. No reported result on Lacuna covers distilled students under 3B on any of these benchmarks, so all student numbers will be new.
Three outcomes would settle the proposal. The first is the mechanism test. On naive students, hallucinated objects should have markedly lower grounding margins than correct ones, and their text-only probability should rise with corpus co-occurrence. If the margins do not separate hallucinated from correct mentions, the core claim fails regardless of downstream numbers.
The second is the inheritance test. The share of mentions with near-zero margin should grow as the student shrinks, and should exceed the teacher's. Without this, distillation is not the source of the problem.
The third is the intervention test. Counterfactual-pair distillation should raise grounding margins, improve adversarial POPE F1 and sentence-level CHAIR beyond both cleaned-corpus and rebalanced distillation, and reach the Owl-decoded student in a single pass, without lowering recall or caption length. As a secondary memorization check, it should also lower the verbatim continuation rate and caption membership AUC. A drop in hallucination without any drop in text-side memorization would mean the method works for a reason other than the one we claim.
References
- Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs
- Membership and Memorization in LLM Knowledge Distillation
- Beyond Modality Collapse: Representation Blending for Multimodal Dataset Distillation
- Quantifying Cross-Modality Memorization in Vision-Language Models
- Evaluating Object Hallucination in Large Vision-Language Models
- Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
- Hallucination is a Consequence of Space-Optimality: A Rate-Distortion Theorem for Membership Testing
- Mitigating Object Hallucination via Data Augmented Contrastive Tuning
- ModDrop: adaptive multi-modal gesture recognition
- Multi-Object Hallucination in Vision Language Models