Keeping Only Reasoning Traces That Need the Image

Models that reason over images, such as charts, geometry figures and diagrams, are increasingly trained on large sets of worked solutions produced by a stronger "teacher" model. These datasets are cleaned mostly by checking that the final answer is right. A correct answer, however, says little about whether the solution actually used the picture. A trace can reach the right number by reading the chart title, guessing from the question wording, or asserting a coordinate it never looked up. Training on such traces plausibly teaches the student to reason without looking. This matches the "seeing is not understanding" failures reported across the field.

Machine translation faced the same problem a decade ago. Web-mined sentence pairs often have a fluent target sentence that does not correspond to its source. The field's answer was to score each pair by how much the target actually depends on the source, and to discard pairs that fail. We propose the multimodal analogue. We score each reasoning step by how much more likely it becomes when the model sees the real image instead of a damaged one. We then keep traces whose steps genuinely depend on the image, and select the kept set so that it still covers many kinds of visual structure.

Core Claim

For a question $q$ with image $I$ and solution steps $s_1,\dots,s_n$, define the visual dependence of a trace as

$$G(x) = \frac{1}{n}\sum_{t=1}^{n}\Big[\log p_\theta(s_t \mid s_{<t}, q, I) - \log p_\theta(s_t \mid s_{<t}, q, \tilde I)\Big],$$

where $p_\theta$ is a fixed small scoring model and $\tilde I$ is a corrupted image: blanked, with its text regions masked, or swapped for another image of the same diagram type. $G$ is the average number of nats each step gains from seeing the true image. A trace whose steps are just as likely without the image is a shortcut trace, even if its answer is correct. A step that states a visual fact, such as a number or label, but gains nothing from the image is flagged as a likely ungrounded assertion.

The claim is twofold. First, at a fixed data budget, selecting training traces by $G$ yields a model that is more robust to perceptual shortcuts than selection by difficulty. Second, it does so without losing headline accuracy on visual math benchmarks. The mask-the-text-regions corruption matters most, because it directly targets the title-reading shortcut.

Where This Sits

Three papers establish the problem. The MATH-Vision study found that 42.2% of GPT-4V's errors are reasoning errors made after correct recognition (Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset). Plot Twist found that chart accuracy reaches 79% when the title contains the answer and falls to the 30% range when the title is removed or misleading. It also found that models do worse on elementary reads than on complex ChartQA questions for the same images (Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details). Together these say that answer-level success hides missing grounding. That is exactly the failure an answer-correctness filter cannot see.

The methods we build on come from translation filtering. Web-mined corpora are highly non-uniform: the top LASER-ranked 25k pairs can match human-curated data, while the bottom 250 En-Si WikiMatrix pairs were only 0.4% correct (Quality Does Matter). The NRC WMT19 system combined cross-lingual correspondence scoring with diversity-aware reranking (NRC Parallel Corpus Filtering System for WMT 2019). The AFRL system scored pairs with a seed translation model trained on clean data. It then showed that quality alone or coverage alone underperforms joint selection with a submodular coverage term, a coverage measure with diminishing returns for repeated content (Quality and Coverage: The AFRL Submission). We reuse both moves. A small seed model scores source-to-target dependence, here image-to-trace, and a coverage term over visual types keeps the kept set diverse.

The closest multimodal work differs in specific ways. MMFineReason filters 1.8M distilled traces by template, length, deduplication, correctness and difficulty. It finds that a 123K difficulty-selected subset matches the full set, but the data ends up 79.4% mathematics (MMFineReason). Its filter never asks whether the trace used the image. Plot Twist's paradox also suggests that difficulty filtering may discard exactly the elementary perception items models most lack.

UniFilter trains a small multimodal classifier on synthetic graded negatives to give one holistic quality score per image-text pair (Train a Unified Multimodal Data Quality Classifier with Synthetic Data). It targets captions and web documents rather than step-level reasoning, and it measures plausibility rather than dependence on the image.

VTPerception-R1 rewards consistency between a written perception scratchpad and later reasoning during online reinforcement learning, with a reported +4.6 on MathVista over its fine-tuning stage (VTPerception-R1). It checks the trace against the model's own description, not against a counterfactual image, and it does not operate on offline data selection.

CPQS-Tuning shows that quality is model-specific and can be read from the target model's hidden states, in text-only settings (CPQS-Tuning). This constrains our design: we must test whether the scorer should be the target model itself.

Research Program

We start from the public MMFineReason pool, whose traces are already answer-verified. This means every trace we score has passed the correctness filter, and we isolate what correctness misses. A 2B-parameter open vision-language model acts as the seed scorer, in the AFRL role. It computes $G$ and per-step gains under three corruptions: blank image, text-region mask using an off-the-shelf OCR detector, and same-type image swap. The cost is a few forward passes per trace, with no generation and no API teacher.

Before any training, we validate the score against people. Annotators label 600 traces, stratified by source dataset, as grounded, partly shortcut, or shortcut, and mark ungrounded visual assertions at the step level. We report how well $G$ ranks shortcut traces below grounded ones, compared with three alternative scorers: a UniFilter-style classifier, a large-model judge prompt, and a text-only "blind solvability" flag.

Selection then maximizes the sum of $G$ over chosen traces plus a submodular coverage term over visual-primitive categories. These categories are chart type, geometric structure, puzzle type and elementary-read versus multi-step, taken from source-dataset metadata and a lightweight tagger. We use AFRL's greedy algorithm, with the budget fixed at 123K to match MMFineReason's subset. Finally, we fine-tune the same 4B base model on each selected subset with an identical recipe and compare.

Baselines

The naive reference point is a random 123K subset of the answer-verified pool, alongside the full 1.8M pool. The random subset tells us whether any selection matters at this budget. The full pool tells us whether selection matches it at 7% of the cost.

The strongest existing method is MMFineReason's own difficulty-aware 123K subset, which its authors report matches the full dataset. We will use their released subset if available, and otherwise re-implement their filter on the same pool. Either way we train with the same base model, epochs, learning-rate sweep and token budget as our subsets. No reported result was found on Lacuna for the absolute MathVista or MATH-Vision accuracy of MMFineReason's 123K subset, so we measure it ourselves. As external anchors we use GPT-4V's 49.9% on MathVista (MathVista) and GPT-4o's 30.39% on MATH-Vision.

Beating this baseline means one of two outcomes. The first is at least matching it on MathVista and MATH-Vision (within seed-to-seed noise, three seeds) while clearly improving Chart101 elementary accuracy and the title-removed or misleading-title condition. The second is a strict improvement on the benchmark averages.

A holistic-quality baseline selects the top 123K by a UniFilter-style scorer applied to question, image and trace, with the same coverage term. This separates "dependence on the image" from "looks like good data."

A question-level baseline drops every question a text-only model answers correctly, then samples randomly from the remainder. Finally, two ablations isolate our ingredients. The first uses $G$ without the coverage term. The second recomputes $G$ with the target 4B model as scorer instead of the 2B seed model, to test CPQS's model-specificity finding.

Why not just…?

Why not just filter by difficulty, as MMFineReason already does? Difficulty and grounding are different axes. Plot Twist shows models can score higher on "hard" chart questions precisely because they shortcut, so a difficulty filter may keep shortcut traces and drop elementary perception items. The equal-budget comparison, read on Chart101 elementary and title-ablated accuracy, settles this directly. If difficulty selection matches us there too, our claim fails.

Why not just drop questions a blind text-only model can answer? This is the most serious threat, because it is the cheap question-level version of our idea. It cannot catch a trace that shortcuts on a question which genuinely needs the image, such as reading an answer from a title the corruption masks. Nor can it catch a correct trace that asserts an unlooked-up coordinate. The blind-solvability baseline at equal budget tests whether that trace-level residue matters. If it matches $G$, we will report that the simpler filter suffices.

Why not just use a learned quality judge like UniFilter, or embedding similarity like CLIPScore? UniFilter already outperformed CLIP-based filtering on its data. But a holistic judge rewards fluent, plausible solutions, and shortcut traces are often fluent. The human-validation ranking comparison and the holistic-quality training baseline measure whether plausibility and image dependence come apart in practice.

Why not just train with grounding rewards online, like VTPerception-R1? Online reinforcement learning is far more expensive per sample, and its reward checks the model against its own description. Offline selection is complementary rather than competing: a better fine-tuning starting set should help any later reward stage. We will test this with a small reinforcement-learning run from each fine-tuned checkpoint, if compute allows.

Proposed Experiments

The first decisive check happens before any training. $G$ must rank human-labeled shortcut traces below grounded ones better than the three alternative scorers. If it does not, the pipeline stops there.

The second check is the stress test from Plot Twist. The evaluation compares each trained model's Chart101 accuracy with the title shown, removed and misleading. Our hypothesis is that $G$-selected training shrinks the gap between these conditions, which currently runs from 79% to the 30% range for tested models, more than any baseline does. This must happen at no cost on MathVista, MATH-Vision and GAOKAO-MM, with the last used as an out-of-distribution check since none of its sources are in the pool.

The third check is per-corruption: which of blank, text-mask or swap drives the gains. It also tests whether the coverage term lowers the mathematics share below 79.4% while holding benchmark accuracy.

A secondary measurement tracks GSM8K before and after tuning. The mitigation paper reports that multimodal tuning degrades text math (Training-Free Mitigation of Language Reasoning Degradation). We ask, as an open question rather than a claim, whether grounded-trace selection changes that degradation.