Evidence-Checked Distillation for Reading Table Images

Large vision-language models can often answer a question about a table image correctly while looking at the wrong part of the page. On a document benchmark with bounding-box labels, GPT-5 answers 81.45% of questions correctly, but the box it points to as evidence overlaps the true evidence region by only 0.90% on average (BBox DocVQA). The model is guessing from language priors and getting lucky. This matters because the standard way to build a small, cheap table-reading model is to have a large "teacher" model write worked solutions, keep the ones with the correct final answer, and fine-tune a small "student" on them. That filter cannot see the problem above: correct answers reached for the wrong reasons pass straight through, and students are known to copy their teachers' systematic errors as if they were fact (The Curious Case of Hallucinations in Neural Machine Translation, Okay, Let's Do This!).

We propose to filter synthetic solutions by whether their cited evidence actually produces the answer, rather than by whether the answer is right. A teacher solution survives only if a separate reader can reproduce the answer from a high-resolution crop of the cited cells alone, and the teacher fails once those cells are blanked out. The same signal also addresses a second problem. Document images expand into thousands of visual tokens (the patch embeddings a vision encoder feeds the language model), and a student trained to name its evidence cells first can then spend its token budget only on those cells. In two sentences: the "right answer, wrong place" failure of teacher models is the dominant bias that small table-image readers inherit, and a check on whether the evidence alone yields the answer removes it. A student trained on the surviving data learns to locate and then read, which also makes it cheaper than pruning methods that ignore table structure.

Core Claim

Our claim has three parts. First, filtering teacher rationales on answer correctness alone transmits an "ungrounded shortcut" behavior to students, and this is measurable as low evidence overlap and a high rate of unchanged answers when the true evidence is erased. Second, a two-role check (a locator that cites cells, and a reader that sees only the cited crop) removes most of that inheritance at matched data size, without lowering answer accuracy. Third, a student trained this way can run a cheap low-resolution pass, predict cell boxes, and re-encode only those boxes at high resolution. At equal visual-token budgets, this beats query-similarity pruning on tables.

How This Relates to Prior Work

Two papers motivate the gap. BBox DocVQA documents the gap between answer accuracy and grounding. It also shows that full-page input loses up to 25% accuracy relative to pre-cropped regions, but it builds a dataset and does not use grounding to filter training data or to choose tokens. TabComp shows that OCR fails on merged-cell and handwritten industrial tables, and that fine-tuned OCR-free readers produce ungrounded, repetitive answers. Together these constrain us to stay in pixel space.

Three works supply pieces we reuse but combine differently. Reflective Instruction Tuning also filters synthetic vision rationales, using a second model's consistency judgment. Its +12.7 POPE gain shows filtering helps, but the judge shares the teacher's priors and sees the whole image, whereas our reader is deliberately denied the full page. MATSA uses multiple agents to attribute table answers to rows and columns. It converts images to HTML first and never distills, whereas we keep evidence as image regions and use attribution as a training filter. TART distills GPT-4 table programs into 7B students (+41.9% over chain-of-thought) on text tables only. It does not measure what students inherit and names raw table images as future work.

On the multi-agent side, Table-Critic and MACM show that separating roles cuts error propagation. For example, Table-Critic's solution degradation is 0.7% versus 4.9% for single-thread Critic-CoT. MACM also suggests using agent systems to generate verified data for small models. We take that suggestion, but our role split is motivated by restricting what each role can see, not by critique. On token reduction, TRIM keeps patches similar to the prompt under CLIP (an image-text matching model) and cuts 576 tokens to about 123 on natural images. VASparse shows that pruning can even reduce hallucination. Neither was tested on tables, where the decisive cues are grid lines, decimal points, and unit markers that carry little semantic similarity to the question.

Research Program

The program has three parts.

Training corpus with exact cell coordinates. We render WikiTQ, TabFact, and the table portions of TAT-QA and FinQA into images under varied styles: fonts, borders, merged headers, scanned noise, and unit footnotes such as "$ in millions." Rendering gives exact cell coordinates for free. We add the real documents of BBox-DocVQA and TabComp, which reflect real-world layouts.

The teacher pipeline. It has two roles with separated context.

  • Locator. A large vision-language model sees the full page. It outputs a solution, a list of cited cell boxes, and, for arithmetic, a short Python program over the cited values. This follows PoTable's lesson that explicit type cleaning prevents silent numeric failures (PoTable).
  • Reader. A separate call receives only the high-resolution crops of the cited boxes plus the question. It must reproduce the cell values, and the program run on those values must give the locator's answer.

A trajectory is kept only if it passes two tests. It must pass this sufficiency test. It must also pass a necessity test: the locator, shown the page with the cited region blanked, no longer produces the same answer. We run two teachers, GPT-5-class and Qwen2.5-VL-72B, because they differ sharply in grounding (0.90% versus 35.20% mean IoU on BBox-DocVQA). That difference lets us study whether students inherit the specific pathology of their teacher.

The student. A 7B-scale vision-language model such as LLaVA-1.5 is trained to emit boxes and then the answer. It is not asked to run agents at inference, since models under 13B do not hold multi-role protocols reliably (Self-collaboration Code Generation). At inference it encodes the page at low resolution, predicts boxes, and re-encodes only those crops at full resolution. Its token count is set by how much evidence it cites, not by page size.

Measuring inheritance. We use three measures:

  • evidence IoU against gold boxes;
  • the shortcut rate, the fraction of questions where the student gives the same answer with the gold evidence erased;
  • teacher–student error correlation, i.e., whether the student fails on the same questions as its teacher, beyond what difficulty explains.

On rendered tables we can also audit the filter itself, measuring how often it accepts trajectories whose cited cells are wrong. This addresses the worry that a model-based filter shares its teacher's blind spots.

Baselines

Zero-shot student. The naive reference is the untrained 7B student applied zero-shot at full resolution. It tells us how much any distillation helps.

Answer-correct filtering. The strongest competitor on the inheritance question is the same teacher data filtered only by final-answer correctness. This is the standard rejection-sampling recipe implicit in TART. It is the right foil because it is cheaper and may yield equal accuracy. We train it on exactly as many examples as survive our filter, sampled from the larger answer-correct pool, with the same student, steps, and hyperparameter search. Beating it means matching or exceeding its answer accuracy while substantially raising IoU and lowering the shortcut rate. If accuracy and shortcut rate are both unchanged, our claim fails.

Judge filtering. A second filter baseline is a consistency judge in the style of Reflective Instruction Tuning. The same large model, seeing the full page and the rationale, accepts or rejects. It also runs at matched data size. Comparing against it isolates the effect of denying the full page to the checker.

Ablations of the check. One ablation gives the reader the full page instead of the crop. Another drops the necessity test. Together they show which half of the check carries the effect.

Teacher references. On BBox-DocVQA, GPT-5 reports 81.45% accuracy with 0.90% mean IoU, and Qwen2.5-VL-72B reports 35.20% mean IoU (BBox DocVQA). We expect our student to fall below GPT-5 in accuracy but above both teachers in IoU. A useful result is IoU above 35.20% at 7B scale.

Upper references. For the rendered WikiTQ and TabFact splits, the text-input multi-agent system Table-Critic reports 73.7% and 91.7% accuracy. This serves as an upper reference for what perfect reading would allow, not a target. On TabComp, fine-tuned Donut-base reports BLEU-4 42.69 and BERTScore 83.38 (TabComp). On DocVQA, the original best baseline reports 0.665 ANLS (average normalized Levenshtein similarity) (DocVQA).

Efficiency baselines. For the token question we compare against uniform downsampling and TRIM applied to the same student at matched token budgets. No reported result found on Lacuna for TRIM or VASparse on table or document benchmarks, so we run both ourselves.

Why not just…?

Why not just filter on answer correctness, which is cheap and already works for TART? This is the real threat. If most correct teacher answers on tables happen to be grounded, the extra check buys nothing. The BBox-DocVQA numbers suggest otherwise, at least for GPT-5. However, they come from a localization task, not from rationales, so the inference is not proven. The matched-size comparison above settles it directly. If the answer-filtered student shows the same shortcut rate as ours, we drop the claim.

Why not just OCR the table to Markdown and use a text model like Table-Critic? On clean rendered tables this may win on accuracy, and we report it as an upper reference. TabComp shows OCR failing on merged cells and handwriting, and TAT-QA attributes 55% of model errors to evidence extraction even on text (TAT-QA). We therefore evaluate the comparison separately on real scanned documents, where we expect the OCR pipeline to lose.

Why not just use an off-the-shelf pruner like TRIM? TRIM ranks patches by semantic similarity to the question. Grid lines, decimal points, and unit footnotes resemble no question word, yet a misread unit flips an answer. Our accuracy-versus-tokens curves on rendered tables, where we control whether a unit marker or border is decisive, show directly whether similarity pruning drops these cues.

Why not just a single model that cites and checks itself? Self-checking with the full page in view lets the model reuse the same prior that produced the shortcut. Table-Critic's degradation numbers suggest that single-thread reflection is unreliable. The full-page-reader ablation tests this directly. If it matches the crop-only reader, the separation of roles is unnecessary and we say so.

Proposed Experiments and Decisive Checks

Inheritance. The central experiment crosses two teachers with three filters (answer-only, judge, and our evidence check) at matched data size. We evaluate on rendered WikiTQ, TabFact, and TAT-QA images, BBox-DocVQA, TabComp, and DocVQA. We report accuracy (or ANLS, BLEU-4, and BERTScore where those are the benchmark metrics), evidence IoU, shortcut rate, and teacher–student error correlation. The decisive pattern is twofold. First, students under answer-only filtering should show shortcut rates that track their teacher, higher for the GPT-5-class teacher than for Qwen. Second, our filter should collapse that difference.

Filter precision. On rendered tables, we measure filter precision against gold coordinates. This shows whether a model-based checker still lets wrong-evidence trajectories through.

Efficiency. We plot accuracy against visual tokens per page for full resolution, uniform downsampling, TRIM, and locate-then-read. We use a stress split in which the deciding information is a non-semantic mark: a unit line, a minus sign, or a merged-header boundary. The hypothesis is confirmed if locate-then-read dominates the curve on this split and roughly ties on easy questions. It is refuted if TRIM matches it at equal budgets.

Human-gold reference. A small human-gold training subset, drawn from BBox-DocVQA's verified boxes, gives a reference for how close filtered synthetic data comes to the best supervision available.