Vision-language models (systems that read an image and answer questions about it in text) often reach the right answer about a chart or diagram for the wrong reason. They may read the answer off the chart's title, or reason from a number they never actually extracted from the plot. Their written step-by-step explanations then justify the answer after the fact. The usual remedies are costly. Checking a multi-step derivation by hand takes a lot of human effort, and automated verifiers can hallucinate and anchor on the answer just as the reasoner does. This proposal rests on a simple asymmetry. Asking a person "does bar 3 in this chart reach about 42?" is a two-second yes/no glance. Asking them to grade a whole derivation is a slow expert task. If a model is made to state the visual facts it relies on as short, checkable claims, people (or synthetic ground truth) can verify those facts cheaply. Editing the image so that a claimed fact changes then tests whether the answer actually depends on it.
The result is a training signal that rewards a reasoning trace for two things. Its stated visual premises must be true, and its answer must move when those premises are changed in the pixels. We use this signal to build preference data for an open vision-language model on chart and visual-math benchmarks. We test whether the signal removes title shortcuts and decorative reasoning without the accuracy loss that faithfulness steering causes in text models.
Core Claim
Post-hoc rationalization in multimodal reasoning is a problem of ungrounded premises. It can be measured and corrected at the level of premises rather than reasoning steps, and the two required signals are both cheap. The first is premise truth. It comes from one-bit human checks of the kind used in One-bit Supervision for Image Classification and Test-Time Adaptation with Binary Feedback, or for free from synthetic charts. The second is premise causality. It comes from re-rendering the image with the premise's value changed and checking whether the model's answer follows. We claim that preference training on traces that pass both tests improves robustness to shortcut removal more than outcome-only training, text-only causal training, or automated step verification. We also claim it does so at a human cost per item that is a small fraction of full-trace grading.
What is new is where the intervention happens. FRIT measures a step's causal importance by swapping text steps for distractor facts. It is purely textual, evaluated on GSM8K, SVAMP and StrategyQA. In a multimodal trace this misses the main failure, because the shortcut lives in the image, for example in the title, and editing the text trace leaves it intact. MM-Verify trains an automated 7B verifier with tree-search rollouts. It never asks whether the verifier or the reasoner is causally faithful, and it never involves human checks. Bootstrapping Grounded Chain-of-Thought (GCoT) appends verified bounding boxes to reasoning traces. It measures only accuracy, so its boxes could themselves be decorative. Its boxes also fail on curves, angles and trend lines, which our value-level premises can express.
Research Program
The motivating evidence comes from three diagnostic papers. Plot Twist shows that models answer complex ChartQA questions better than elementary ones on the same charts. It also shows that accuracy falls from 79% to the 30% range when a title that gives away the answer is removed. MATH-Vision attributes 42.2% of GPT-4V's errors to reasoning and 31.9% to perception, and finds that chain-of-thought prompting sometimes lowers multimodal accuracy. Can Aha Moments Be Fake? finds that most text reasoning steps have no causal effect on the answer. On the Hardness of Faithful Chain-of-Thought Reasoning constrains the design: steering text models toward faithfulness cost up to 20% accuracy. Avoiding that trade-off is therefore something we must show, not assume.
Premise-structured traces. We prompt the base model, and later fine-tune it, to open each trace with a short premise block before any derivation. Each premise is an atomic visual claim of the form element, attribute, value, such as "line B at x=2010, y ≈ 3.1" or "angle ABC ≈ 40°". Free derivation text follows. This borrows the core move of skeleton-guided reasoning from Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation, which separates structure from content. Here the separation is between visual claims and derivation. That paper's finding that telling a model to suppress the answer backfires is our reason for not simply instructing the model to "ignore the title."
Two cheap tests per premise. The truth test asks whether the claim is correct. On synthetic charts we generate in the style of Chart101, the answer comes from the underlying data. On real images (ChartQA, MathVista, MATH-V), a human answers yes or no while viewing the image with the referenced element highlighted, much as Error Span Annotation replaced error taxonomies with simple marks. We send to humans only the premises where the model is least certain about its own claim. That routing follows BiTTA's uncertainty-based querying, though we use agreement among sampled traces instead of Monte Carlo dropout, which is expensive. The causal test applies to synthetic charts. We re-render the chart with that premise's value changed so that the correct answer changes, and we also re-render with the title removed or made misleading. A trace counts as faithful if its premises are true, its answer tracks the edited value, and its answer is unaffected by title changes that do not alter the data.
Training. For each question we sample several traces. Traces that pass both tests are preferred over traces that reach the same final answer but fail either test. That pairing isolates faithfulness from correctness. We then train with direct preference optimization (DPO, which fits a model to pairwise preferences without a separate reward model), as FRIT does. The base model is an open 7B-scale vision-language model of the kind used as a reasoner in MM-Verify. Causal tests run only on synthetic charts. Transfer to real charts and diagrams, where only truth checks are available, is a central empirical question.
Baselines
The naive reference is the base model with ordinary step-by-step prompting and with direct answering. MATH-Vision reports that the two can rank in either order, so we report both.
The strongest accuracy competitor on MathVista is MM-Verify's reasoner plus verifier, reported at 65.3 accuracy with 12 rollouts versus 63.8 for GPT-4o (MM-Verify). We reimplement its verifier-guided rejection sampling on our base model, using the same training questions and the same number of sampled traces per question. We then train with the same DPO setup, so only the source of the preference signal differs. On ChartQA the strongest reported system is ChartPaLI-5B with Program-of-Thought at 81.3% (Chart-based Reasoning). That system uses different pretraining and parameter counts, so we cite it as context rather than as a matched baseline. A fair head-to-head comparison would require its chart-to-table pretraining, and we discuss that below.
The closest method competitor is FRIT applied directly to our multimodal traces: distractor substitution on text steps, followed by DPO. It gets the same sample and compute budget. An outcome-only baseline prefers correct-answer traces over incorrect ones with no premise signal. This is the obvious cheap alternative.
Two ablations isolate the key ingredients. One uses premise truth only, with no image re-rendering. The other uses the causal test only, without truth checks. A third arm keeps the full method but replaces the human yes/no checks with the MM-Verify-style verifier's judgments of the same premises.
Beating a baseline means two things. On the shortcut-robustness metrics below, we must exceed every matched baseline by more than the variation across three random seeds. On standard accuracy, we must not fall below the outcome-only baseline by more than one point. No reported result was found on Lacuna for any causal-faithfulness metric on multimodal reasoning, so all faithfulness numbers will be produced here.
Why not just…?
Why not just train on correct answers (outcome reward)? Plot Twist shows that correct answers on charts are often produced by title shortcuts, so rewarding correct answers rewards those shortcuts. The outcome-only arm tests this directly. If it matches our method on title-removed and value-edited charts, the premise machinery is unnecessary, and we would report that.
Why not just use an automated verifier instead of people? This is the most serious threat. MM-Verify's verifier scores 61.7 on MathCheck, so it is useful but far from perfect. Its authors also note that intermediate model rewards hallucinate. Atomic visual premises may, however, be much easier for a model to verify than whole traces. The verifier-oracle ablation settles the question. We compare the agreement of automated premise labels with synthetic ground truth against human agreement, and the downstream gain from each. If the automated verifier is within noise of the human checks, the contribution becomes the premise and image-edit structure itself, and humans are needed only for auditing.
Why not just derender charts to tables and compute (Chart2Table plus Program-of-Thought)? ChartPaLI's results show that this works well when an underlying table exists. It needs synthetic tables, though, and it discards spatial and geometric information. It also does not apply to MATH-V geometry or folding problems. We therefore evaluate on MATH-V and the geometry portions of MathVista, where no table exists. We also report a derender-then-compute pipeline on charts, to show where our method adds nothing there.
Why not just apply FRIT as is? Text substitution cannot touch a shortcut that sits in the pixels. The FRIT arm, scored on title-edited charts, measures exactly how much this matters.
Proposed Experiments
The decisive test is shortcut robustness on held-out synthetic and Chart101-style charts. We measure accuracy under three conditions: an answer-revealing title, a removed title, and a misleading title. The quantity of interest is the gap between them, which is 79% versus the 30% range in Plot Twist. Alongside it we report Chart101 collective accuracy, the fraction of charts on which every elementary question is answered correctly, which is below 25% for top models in Plot Twist. Our main faithfulness metric is counterfactual tracking: the fraction of re-rendered charts on which the answer changes exactly as the edited data requires. We also report early-answering faithfulness from Tanneru et al., adapted by truncating the premise block. Transfer is measured on ChartQA, MathVista and MATH-V accuracy, including the MATH-V arithmetic-versus-transformation-geometry split. Here the question is whether accuracy rises, as FRIT found for text, or falls, as faithfulness steering did on TruthfulQA.
A human study measures cost. Annotators either answer the routed yes/no premise checks or grade full traces, the latter as a marking-only grading pass in the spirit of ESA. We compare median seconds per item, agreement between annotators, and the downstream gain per annotator-hour.
The hypothesis fails if counterfactual tracking does not exceed the outcome-only and FRIT arms, or if transfer from synthetic causal tests to real benchmarks is absent. It also fails if per-item human cost does not fall well below that of full-trace grading.
References
- One-bit Supervision for Image Classification
- Test-Time Adaptation with Binary Feedback
- FRIT: Using Causal Importance to Improve Chain-of-Thought Faithfulness
- MM-Verify: Enhancing Multimodal Reasoning with Chain-of-Thought Verification
- Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptation
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Can Aha Moments Be Fake? Identifying True and Decorative Thinking Steps in Chain-of-Thought
- On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models
- Measuring and Mitigating Post-Hoc Rationalization in Reverse Chain-of-Thought Generation
- Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation
- Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs