Multimodal models answer fine-grained visual questions better when someone first marks the relevant region on the image, for example with a red circle, a blurred background, or a soft heatmap, while leaving the rest of the picture intact so spatial relations and scene context survive. These markers are slow to produce. They need an extra model pass or an auxiliary search, about 4 seconds per query for one published pipeline. The standard fix is to distill: a large teacher produces focus regions offline, and a small student learns to place its attention there internally in a single pass. This proposal asks what the student learns from that training. Our hypothesis is that it copies the teacher's mistakes about where to look, and that this goes unnoticed because the usual quality filter (keep a teacher example only if its final answer is right) lets through many examples where the teacher looked in the wrong place and still guessed correctly.
We will test this with a benchmark that is generated procedurally and rebuilt for every evaluation. In each item, the region the question depends on is known exactly because the scene was rendered from a program. The scene can then be edited so that the true region and the teacher's likely prior disagree, for instance by moving the target or adding a salient decoy. On these scenes we measure how often student focus errors follow teacher focus errors. We then compare answer-only filtering with a validator that keeps a teacher's focus region only if hiding that region changes the answer and hiding a matched control region does not. The study connects four threads. Visual prompting supplies the mechanism being distilled. Inference latency is the reason distillation is needed. Bias inheritance is the scientific question. Contamination-proof generation is what makes the measurement trustworthy.
Core Claim
When a context-preserving focus signal is distilled into a single-pass student, the student's localization errors are strongly conditioned on the teacher's localization errors. This inheritance is concentrated on items where the teacher's prior (visual salience, typical object position, textual cues) points away from the true region. Answer-correctness filtering removes only a small part of it. A causal masking validator, which tests the teacher's region against the image instead of asking another model's opinion, should remove much more, at no inference-time cost. The claim is falsifiable. If student focus errors are statistically independent of teacher focus errors once answers are filtered, inheritance is not a practical problem, and we will report that.
Where This Sits
The mechanism we study comes from context-preserving visual prompting. Three papers show that marking a region on the full image beats cropping it away because relational context survives: What does CLIP know about a red circle? uses drawn circles, Fine-Grained Visual Prompting blurs the background, and Spot-IT blends a query-guided Gaussian heatmap into the image. DCoT has the model draw its own boxes in a first dialogue turn. It notes that a wrong box in that turn cannot be recovered from. That observation is the per-query version of the failure we expect to see baked into weights after distillation.
Two papers already compress focus into a fast student, and they are the closest prior work. SceneDiver distills coarse-to-fine focus plans into a slot-attention adapter that adds about 3 ms. Latent Visual Reasoning trains Qwen2.5-VL to emit hidden states that reconstruct the embeddings of query-relevant image patches; it uses a fixed token budget because a learned stopping token proved unstable. Both report answer accuracy on static benchmarks. Neither asks whether the focus targets they trained on were correct, or whether the student reproduces their errors. We reuse their student designs and audit what they inherit.
The inheritance hypothesis is motivated by evidence that distillation amplifies teacher pathologies. The Curious Case of Hallucinations in Neural Machine Translation shows this for translation hallucinations, and Okay, Let's Do This! shows students adopting hallucinated rationales. DocVAL is the strongest constraint on our design. It shows that teacher traces in document question answering contain plausible but spatially wrong boxes, and that rule-based validation gives +6.3 mAP over binary answer filtering. Its validator relies on OCR text detection, however, and fails on charts, diagrams, and handwriting. Our masking validator needs no detector. Reflective Instruction Tuning fights inherited shortcuts by supervising with negative rationales and cross-checking them with a model. The file's own caution applies here: filters built from secondary models share the generator's blind spots. Our validator intervenes on pixels for exactly that reason.
The evaluation design draws on contamination work. MATHVERSE finds that some models score 5%+ higher with the diagram removed, so static visual benchmarks can reward ignoring the image. DyVal and A²RBench generate items programmatically; A²RBench reports a 17.7% drop under symbol remapping and also a warning we take seriously: when a language model writes the generator, it simplifies hard spatial logic. We therefore write the scene renderer and question templates as fixed code, not model-authored programs. Beyond Direct Gains supplies the matched-control idea: format-only, helpful, and misleading scaffolds. We carry it over to visual markers.
Research Program
The benchmark: a procedural focus audit. A scene program places objects, text snippets, and chart or document elements on a canvas. A question template then picks a relational query whose answer depends on one region and its surroundings, such as "what is written on the card to the left of the blue mug." The answer and the minimal relevant region are known exactly. For each base scene we generate counterfactual siblings:
- a target-moved variant, which relocates the relevant region to an unusual position;
- a salient-decoy variant, which adds a high-contrast distractor;
- a text-cue-stripped variant, in which the question carries no lexical hint toward the region, following MathVerse's decomposition;
- a remapped variant, which uses novel object names and colors, following A²RBench.
The prior-conflict split is the set of items where a teacher's typical bias (salience, central position, text density) points away from the true region. Items are regenerated with fresh seeds at each evaluation, so no fixed test set can leak into training.
Teachers and students. We use two teachers. The first is Spot-IT-style heatmapping on a large VLM. The second is DCoT-style self-generated boxes. Each produces focus regions on a training pool that mixes natural images with procedural scenes; the procedural scenes in that pool are disjoint in seed and template from evaluation. We use two students. The first is a Qwen2.5-VL-3B/7B model trained with the Latent Visual Reasoning objective, regressing teacher-region patch embeddings under a fixed latent budget. The second is a SceneDiver-style adapter. For both, the student's focus can be read out: for the latent student, by nearest-neighbor matching of its emitted states to image patches; for the adapter, from its slot masks.
The validator. For a teacher region $R$ on image $x$, let $f(\cdot)$ be the answer of a fixed reference model under soft masking. We accept $R$ if two conditions hold: $f$ changes when $R$ is blurred out, and $f$ stays the same when an area-matched random region is blurred. We use two different reference models, the teacher itself and an independently trained small VLM, to test how much blind-spot sharing matters.
The central metric. Let $E_T$ and $E_S$ mark a teacher and a student localization error, each defined as IoU with the true region below 0.3. The inheritance gap is
$$\Delta = P(E_S \mid E_T) - P(E_S \mid \neg E_T),$$
computed on held-out procedural items where both teacher and student focus are recorded. If $\Delta \approx 0$, the student's errors are its own. A large positive $\Delta$ means the student copied the teacher's errors. We report $\Delta$ separately on the prior-conflict and non-conflict splits.
Baselines
The naive reference point is the base Qwen2.5-VL with no focus signal, answering directly. It sets the floor for accuracy and latency. Latent Visual Reasoning reports that this base model reaches 66.67% on MMVP.
The strongest existing method for fast focus is the Latent Visual Reasoning student trained on unfiltered teacher regions. It reports 71.67% on MMVP, plus gains of 2.7% on visual detail search and 5.3% on relative spatial reasoning on V* Bench. We reimplement it on the same training images, the same latent budget, and the same number of optimizer steps as every other student. Learning rate is tuned on a shared validation split with an identical search budget per variant. We also build a SceneDiver-style adapter under the same conditions. SceneDiver reports only embodied results (LIBERO-plus, navigation), so no reported number on our benchmarks was found on Lacuna, and comparisons will be against our own runs.
The slow reference is the teacher pipeline applied at inference time. Spot-IT reports +21.05% for GPT-4o on the NiM-Benchmark, a best augmented model at about 46% exact match against about 63% for humans, and about 4 seconds of added latency. This pipeline sets the accuracy the students try to approach at a fraction of the cost.
The key ablation is answer-correctness filtering. It keeps a teacher region only when the teacher's final answer is right, and is the default practice. DocVAL's +6.3 mAP gain over binary filtering is the reported evidence that answer filtering leaves spatial errors behind, although that result is in documents. The validator variants differ from answer filtering only in which examples are kept. To keep the comparison fair, we also match the number of kept examples by random subsampling. Without that control, any gain could come from having less data rather than better data.
On procedural scenes we add two more references. The first is an oracle student trained on true regions, which gives an upper bound. The second is a student trained on deliberately corrupted regions, which gives a calibration point for how large $\Delta$ can get.
The validated student counts as beating the baselines if three things hold together. Its $\Delta$ on the prior-conflict split must fall by at least half relative to the answer-filtered student. Its accuracy on the counterfactual siblings must rise. Its static-benchmark accuracy must stay within noise of the unfiltered latent student, and its latency must stay at single-pass levels.
Why not just…?
Why not just keep the teacher's visual prompt at inference instead of distilling? That is the accurate but slow option, and we run it as the reference. Spot-IT's roughly 4 seconds per query and DCoT's two-turn dependence make it unsuitable for interactive or embodied use, which is why SceneDiver and Latent Visual Reasoning distill in the first place. Prompting at inference also inherits the teacher's errors, just one query at a time. Our counterfactual siblings measure that error directly, so the comparison will show whether distillation makes inheritance worse than prompting does.
Why not just filter teacher examples by answer correctness? This is the most plausible competitor. If it removes most inherited focus errors, our validator is unnecessary. DocVAL's evidence that answer-correct traces still carry wrong boxes points the other way, but only for documents. The size-matched comparison on the prior-conflict split settles the question, and we will report a null result if answer filtering is enough.
Why not just use a rule-based spatial validator like DocVAL's? DocVAL's validator checks boxes against OCR-detected text. Its authors report failures on charts, diagrams, and handwriting, and it has nothing to check against in natural scenes. The masking validator needs no detector. On the document subset we run both validators so they can be compared head-to-head where DocVAL applies.
Why not just train students on human-annotated or detector boxes and skip the teacher? For relational and reasoning questions, the region a question depends on is rarely annotated, and detectors find objects rather than answer-relevant context. That gap is why teachers are used. The oracle student on procedural scenes shows how much is lost relative to true supervision. If the oracle and the validated student are close, validation recovers most of what real annotations would give.
Why not just evaluate on MathVerse's vision-only split or V* Bench? We do, but these benchmarks expose text shortcuts and search difficulty, not where the model looked. They provide no counterfactual sibling and no ground-truth relevant region, so they cannot separate a student that inherited a teacher's error from one that failed independently.
Proposed Experiments and Decisive Checks
The first experiment tests whether inheritance exists. We train the Latent Visual Reasoning and adapter students on unfiltered and answer-filtered teacher regions, then measure $\Delta$ on fresh procedural scenes. The hypothesis survives only if $\Delta$ is clearly positive and larger on the prior-conflict split. We also check whether static accuracy hides it, on MMVP, V* Bench, the NiM-Benchmark, DocVQA, and the MathVerse vision-only versions. This holds if students with similar static accuracy differ sharply in $\Delta$ and in accuracy on the target-moved and salient-decoy siblings. On DocVQA we report ANLS and box mAP against DocVAL's reported 91.4% ANLS and 82.4% mAP for Gemma3-12B, keeping in mind that our students are smaller.
The second experiment applies matched visual controls, adapted from Beyond Direct Gains, at test time. The conditions are no marker, a format-only marker on a random region, a correct marker, and a misleading marker on the decoy. A student that follows the misleading marker as readily as the correct one has learned to obey markers rather than to read the image. We expect unfiltered students to show this more strongly.
The third experiment compares the masking validator against answer filtering at matched data size. We run the validator with the teacher as reference model and with an independent reference model, which tells us how much shared blind spots weaken a pixel-intervention filter. We report latency throughout as wall-clock time per query on the same hardware, so every accuracy gain is weighed against the speed that motivated distillation.
The decisive check is a set of two results that must hold together on the regenerated prior-conflict split. First, the validated student must cut $\Delta$ substantially relative to answer filtering at equal data size. Second, it must not lose accuracy on the remapped and text-cue-stripped siblings. If the first holds without the second, the validator is merely discarding hard examples. If neither holds, inherited focus is not the bottleneck, and fast students' failures should be looked for elsewhere.
References
- What does CLIP know about a red circle? Visual prompt engineering for VLMs
- Fine-Grained Visual Prompting
- Finding Needles in Images: Can Multi-modal LLMs Locate Fine Details?
- DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models
- Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generation
- Latent Visual Reasoning
- The Curious Case of Hallucinations in Neural Machine Translation
- Okay, Let's Do This! Modeling Event Coreference with Generated Rationales and Knowledge Distillation
- DocVAL: Validated Chain-of-Thought Distillation for Grounded Document VQA
- Reflective Instruction Tuning: Mitigating Hallucinations in Large Vision-Language Models
- MATHVERSE: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- A²RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
- Beyond Direct Gains: Matched Controls for Evaluating Concept Scaffolds