Evidence-Pinned Chart Reasoning: Training Answers That Move Only When the Pixels Do

Vision-language models can answer hard questions about charts and still fail at reading the smallest value on an axis. When a chart's title happens to state the answer, they look competent. When the title is removed, accuracy collapses. Rephrasing the question or switching plotting libraries can also flip their answers. This proposal treats these as one failure, not three. The model's answer is not tied to the visual evidence, so anything other than the evidence can push it around: wording, formatting, rendering style, a misleading title.

We propose to generate charts programmatically from known data tables. From each table we build matched twins. Some twins change only nuisances, such as the prompt template, the rendering style, or whether the title gives a hint, and the answer must stay the same. Other twins change one underlying data value, and the answer must change by a predictable amount. A model fine-tuned so that its answers stay put under the first kind of twin and move under the second should become less sensitive to prompt wording and to visual domain shift at the same time. Simply adding varied data cannot ensure this, because a model can learn to tolerate variety without learning to look.

Core Claim

The claim is that prompt brittleness and cross-domain brittleness in chart reasoning share a cause: weak dependence of the answer on the visual evidence. Training that explicitly couples the two directions should therefore fix both more than training that targets either alone. The coupling is invariance to nuisances together with sensitivity to semantic edits.

The motivation comes from three findings. ProSA measures prompt sensitivity in text-only language models. It finds that sensitivity concentrates on instances where the model's output distribution is flat, so trivial wording changes tip the decision. Plot Twist probes chart models with simple questions. It finds accuracy as high as 79% when titles contain the answer, falling to the 30% range when titles are removed or misleading. It also finds that adding grid lines drops PaLI-3 from 29.5% to 21.3%. Agent-ScanKit probes interface-navigating agents. It reports that UI-TARS relies on a text keyword 98.9% of the time to trigger its COMPLETE action, ignoring the screen.

Each of these shows a model answering from something other than the image. None of them proposes a training fix.

Closest Work and How This Differs

Four papers sit closest, and each plays a different role.

Synthesize Step-by-Step supplies the method we reuse. It renders charts from metadata templates and writes executable rationales for them (LAMENDA), which raises MatCha on ChartQA's human-written split from 38.2% to 53.9%. It evaluates only on clean, in-domain charts with fixed templates. It never pairs examples or tests prompt or style shifts.

ChartQA-X generates verified explanations over existing ChartQA images. It likewise neither creates new visual domains nor perturbs prompts.

Reasoning-Driven Multimodal LLM for Domain Generalization motivates the domain-shift side. It shows that embeddings of reasoning chains diverge 58.6% less across domains than raw visual features. That result is for coarse image classification on DomainBed under fixed prompts, and it relies on GPT-4o as a teacher. Our tasks are fine-grained and numeric, and our labels come from the data table, with no teacher model.

CFPO is the closest in mechanism. It penalizes answers that survive when salient visual tokens are suppressed during reinforcement learning. It reports gains of 1.32–2.13% over the previous perception-aware method on ViRL39K, at roughly 35% more time per step and about 10GB more memory, and it breaks fused attention kernels. We build the counterfactual in data space instead: a real re-rendered chart with one edited value. This costs nothing at training time beyond an extra forward pass. It also gives a known target answer rather than only a "should differ" signal.

Finally, Misaligned Roles, Misplaced Images constrains the design. Its adversarial training over 8 structural permutations of the chat template restores safety on Qwen2-VL-7B while keeping VQA-v2 performance stable. Augmentation over prompt structure therefore does work for some goals, which makes it a serious baseline rather than a straw man. That paper never tests multi-step chart reasoning.

Research Program

Generator. We will build a generator that samples a data table and a question family. Question families include elementary reads (axis extremes, a point's value), comparisons, and arithmetic over several marks, written as executable programs in the LAMENDA style so the answer is computed rather than annotated. The generator renders each table under many styles: several plotting libraries, grid lines on or off, marker and color schemes, rotated tick labels, and title conditions (absent, neutral, answer-hinting, answer-contradicting).

Nuisance twins. For each base item, the generator produces nuisance twins that keep the table fixed. These vary rendering, title condition, prompt template, option order and option symbols (following When Benchmarks are Targets), and the placement of the image token and role headers (following the structural perturbations of Misaligned Roles).

Semantic twins. The generator also produces semantic twins that edit one cell of the table. The edit is chosen so that the program's answer changes, and the chart is then re-rendered.

Training objective. Training fine-tunes an open model such as Qwen2-VL-7B. The usual answer loss on every item is combined with two pairing terms. Let $p(\cdot \mid x)$ be the model's distribution over answers for input $x$. For a nuisance pair $(x, x')$, we add a symmetric divergence between $p(\cdot \mid x)$ and $p(\cdot \mid x')$. For a semantic pair $(x, \tilde{x})$ with different correct answers $a \neq \tilde{a}$, we add a hinge penalty when $p(a \mid \tilde{x})$ exceeds $p(\tilde{a} \mid \tilde{x})$. In plain terms, the model is punished for keeping its old answer after the pixels changed.

The objective is

$$ \mathcal{L} = \mathcal{L}_{\text{ans}} + \lambda_n \, \mathrm{JS}\big(p(\cdot\mid x),\, p(\cdot\mid x')\big) + \lambda_s \max\big(0,\; \log p(a\mid \tilde{x}) - \log p(\tilde{a}\mid \tilde{x}) + m\big), $$

where $\mathcal{L}_{\text{ans}}$ is the standard answer loss, JS is Jensen–Shannon divergence, $m$ is a margin, and $\lambda_n, \lambda_s$ weight the two pairing terms. The semantic term is what stops the nuisance term from being satisfied by a model that ignores the image and returns the same answer everywhere.

Analysis. Following ProSA, we will test whether the training reduces sensitivity mainly on low-confidence items. This is the mechanistic prediction of the shared-cause story. We will also test whether the model's answer becomes more often correctly updated under value edits on held-out real charts.

Baselines

The naive reference point is the base model run zero-shot with a single fixed prompt. A fine-tuned variant of it is trained on ChartQA's training split with one template. Both define how brittle the starting point is.

The strongest existing method for chart accuracy is LAMENDA-style synthetic rationale training from Synthesize Step-by-Step. Reported on Lacuna for MatCha, it reaches 53.9% accuracy on ChartQA's human split and 72.6% relaxed accuracy on ChartQA overall. We will reimplement it on our base model, using our own generator restricted to one rendering style, one title condition, and one template. Accuracy is the bar it sets. Beating it means matching its in-domain accuracy while showing lower prompt sensitivity and a smaller drop across rendering styles.

The strongest robustness competitor is unpaired augmentation in the spirit of Misaligned Roles' adversarial training: the same pool of styled, retitled, re-templated, and value-edited items, trained with the answer loss only. This baseline sees every image and prompt our method sees. Only the pairing terms are absent. That makes it the ablation that isolates our key ingredient. On structural permutations, Misaligned Roles reports bringing Qwen2-VL-7B's attack success rate to 0% with stable VQA-v2. No reported result was found on Lacuna for this kind of augmentation on chart reasoning.

Two further ablations separate the two pairing terms: nuisance-only and semantic-only.

A data-space version of CFPO's counterfactual idea is covered by the semantic-only ablation. CFPO itself, run with reinforcement learning on the same items, will be included if compute allows, with its cost overhead reported.

All fine-tuned systems get the same base model, the same number of training items and gradient steps, and the same hyperparameter search budget. The rendering styles and prompt templates used for testing are disjoint from those used in training.

Why not just…?

Why not just augment with paraphrased prompts and varied chart styles? This is the real threat. Misaligned Roles shows that augmenting over structural permutations works for safety. Our unpaired-augmentation baseline is exactly this, at matched data and compute. If it matches the paired objective on prompt sensitivity, cross-style drop, and title-conflict accuracy, the shared-cause claim is wrong. In that case the contribution shrinks to the generator and the factorial benchmark, and we would report it that way. Our prediction is that augmentation lowers sensitivity on nuisances it has seen but leaves the title-conflict and value-edit tests largely unsolved, because nothing in the answer loss distinguishes looking from guessing well.

Why not just convert the chart to a table with an off-the-shelf parser and let a language model reason? ChartQA reports that table-based pipelines fail on questions that refer to visual attributes such as "the orange line." CrossWordBench finds parsing to be the bottleneck, with a correlation of r = 0.9358 between grid-parsing quality and solving quality. A parser trained on one plotting library inherits that library's domain shift. We will run a DePlot-plus-language-model pipeline across our held-out rendering styles to measure where its parsing breaks.

Why not just use few-shot examples or majority voting over several prompts at test time? ProSA finds that one demonstration gives the largest gain in prompt robustness for text models, so test-time anchoring is cheap and partly effective. It cannot fix title shortcuts or grid-line failures, because the demonstrations do not change what the model looks at. We will report one-shot prompting and voting over five templates as test-time baselines, and compare their compute cost per answer.

Why not just use a bigger model? VisualPuzzles finds no clear link between size and reasoning. ProSA finds that strong frontier models can be more prompt-sensitive than smaller ones on hard tasks. We will evaluate a larger model from the same family zero-shot on the full factorial test as a reference point.

Proposed Experiments

Test set. The main test is factorial: held-out tables crossed with held-out rendering styles, the four title conditions, and twelve prompt templates plus the eight structural permutations.

Metrics. We report five quantities on this test. Accuracy is exact match, or within a 5% margin for numeric reads, following Plot Twist. Prompt sensitivity is ProSA's PromptSensiScore across templates. Cross-style drop is the accuracy lost when moving from the training style to held-out styles. Title-conflict accuracy is accuracy when the title contradicts the plot. Evidence-update rate is the fraction of semantic twins whose answer changes to the new correct value.

External benchmarks. We evaluate on ChartQA's human split, PlotQA (where LAMENDA-augmented MatCha reports 92.89%), and Chart101's real and synthetic probes. The Chart101 probes are the closest public test of elementary reading under perturbation.

Out-of-family check. We will measure PromptSensiScore on VisualPuzzles with no training on puzzles. This tests whether reduced brittleness transfers beyond charts, a question no paper in the file has asked.

Decisive checks. The first decisive check is the comparison of the paired objective against unpaired augmentation at matched data. The shared-cause claim predicts that the paired objective wins on title-conflict accuracy and evidence-update rate, and that its advantage in prompt sensitivity concentrates on low-confidence items. If the advantage appears only on nuisances seen in training, or not at all, the claim fails.

The second decisive check transfers to real charts. Using ChartQA images, we hand-edit a small set of charts by changing one bar. The paired model should update its answers on these edits more often than any baseline, without losing human-split accuracy relative to the LAMENDA-style baseline.