Multimodal language models fail at chart and diagram reasoning partly because they cannot see the chart. They also fail because they cannot reason about it, but the seeing part gets less attention. Most of these models attach a pre-trained vision encoder to a language model and keep the encoder frozen. The vision encoder is the network that turns an image into the vectors the language model reads, and "frozen" means it is never updated. Whatever that encoder fails to register (a bar's exact height, which line is orange, where a point sits relative to a gridline) is lost before reasoning begins. Current synthetic-data pipelines respond in two ways. They generate more reasoning text for the language model, and they run expensive verification cascades to throw away generated examples that turn out to be wrong. VeriSciQA keeps only 3.0% of its candidates after checks by GPT-4o, DeepSeek-v3 and o4-mini.
This proposal takes a different route. When a chart is rendered from code, the program already knows every true value, coordinate and color. That metadata can be used directly to train the vision encoder. The training data consists of minimal pairs: two renderings that are identical except for one task-critical visual fact, such as one bar 8% taller, or the same data drawn in a different plotting library. Correctness is guaranteed by construction, so there is no rejection step. Generation budget can also be targeted. A cheap linear probe measures what the encoder currently fails to encode, and generation goes to those failures. The claim is that a modest number of such pairs, used to update the encoder, closes more of the gap on chart reasoning benchmarks than a larger corpus of reasoning instructions passed through a frozen encoder, at a small fraction of the generation cost.
Core Claim
Three bodies of evidence together say that a large share of chart and diagram reasoning failures are perception ceilings inherited from the frozen encoder, and that these ceilings are measurable and fixable with cheap, self-verifying synthetic data.
The motivating evidence is consistent. In MATH-Vision, 31.9% of GPT-4V's errors are vision recognition errors. Plot Twist shows that models read chart titles instead of axes: accuracy reaches 79% when the title contains the answer and falls to the 30% range when it does not. In the same study, adding gridlines drops PaLI-3 from 29.5% to 21.3%, and ChartLlama scores 4.2% on scatter plots. The authors of Multimodal Self-Instruct themselves name a resolution bottleneck in the CLIP encoder of LLaVA-1.5.
One finding constrains the claim rather than supporting it. GeoSQA shows that models still fail when diagrams are fully transcribed into text, and MATH-Vision attributes 42.2% of errors to reasoning. The proposal therefore does not claim that perception is the whole gap. It claims that perception is the part that synthetic pipelines currently leave untouched, and it designs experiments that separate the two kinds of error.
What Is Different From Existing Work
The closest work generates code-grounded chart data but spends the program's ground truth only on supervising the language model.
Multimodal Self-Instruct has a language model propose visual ideas, renders them with Matplotlib and Graphviz, writes question–answer pairs with rationales, and then filters them by code execution, an aesthetic check by a smaller multimodal model, and self-consistency voting. Synthesize Step-by-Step uses SVG chart metadata to teach MPT-7B to break questions into executable Python steps over DePLOT tables. It raises human-written ChartQA from 38.2% to 53.9%. Both pipelines keep perception fixed, and the second inherits DePLOT's extraction errors directly. Neither uses minimal pairs, and neither uses a measurement of encoder blindness to decide what to generate.
Two further lines of work address the problem at test time. BeaconChart runs a planner, a perception agent and an independent verifier on Qwen3-VL-32B. DCoT has the model draw a box around the salient region and re-prompts with it. Both work around the encoder at inference, and neither changes it.
Verification-heavy generators also differ from this proposal. VeriSciQA and InstructDET, which uses CLIP to prune LLaVA hallucinations, pay for correctness after generation. Here correctness comes from construction.
The data-quality lesson is borrowed from text distillation. DISCO shows that 75K label-flipping counterfactuals beat more than 1M unfiltered examples, which suggests that targeted contrast matters more than volume. Its counterfactuals perturb text and are filtered by a teacher model. Ours perturb pixels, and the renderer guarantees the label.
Research Program
The work has three components.
The first is a perception probe. For a frozen encoder (CLIP ViT-L/336 inside LLaVA-1.5-7B, chosen so results are directly comparable to Multimodal Self-Instruct), we render charts with known metadata. We then fit small linear readouts from the encoder's patch features to element-level facts: bar or point value normalized to the axis range, color-to-series assignment, ordering between two marks, and position relative to gridlines. The probe is cheap and gives a per-fact-type, per-chart-style map of what the encoder does and does not encode. We expect it to reproduce the Plot Twist fragilities quantitatively: failures concentrated on scatter plots, gridline styles and non-Matplotlib renderers.
The second is the minimal-pair generator. A sampler draws underlying data tables and rendering specifications across Matplotlib, Plotly, Vega-Lite and Graphviz. Each rendering specification fixes markers, gridlines, and whether the title is informative, absent or misleading. For each base chart the generator emits partners of two kinds. A value-flip partner changes one task-critical fact so that the answer to a templated question changes. A style-flip partner changes only presentation, so the answer must stay the same. Questions and answers are computed from the table by program. A small open model is used only to paraphrase question text. There is no frontier-model call and no verification stage. The only discards are rendering crashes.
The probe allocates the budget. Sampling weight for each fact type and style is proportional to the probe's error on it, so generation concentrates where the encoder is blind.
The third is encoder adaptation. Let $f_\theta$ be the vision encoder with parameters $\theta$ (updated through low-rank adapters to keep compute modest) and $g$ a light readout head. For a pair $(x, x')$ rendered from data whose task-critical values are $v$ and $v'$, we train
$$ \mathcal{L} = \mathcal{L}_{\text{LM}} + \lambda_1 \big(\lVert g(f_\theta(x)) - v\rVert + \lVert g(f_\theta(x')) - v'\rVert\big) + \lambda_2\, \mathcal{L}_{\text{pair}}(x, x'). $$
Here $\mathcal{L}_{\text{LM}}$ is the usual answer-token loss of the multimodal model. The middle term asks the encoder's features to linearly encode the true values. $\mathcal{L}_{\text{pair}}$ pulls together the features of style-flip pairs and pushes apart the features of value-flip pairs in the region the question refers to. The weights $\lambda_1$ and $\lambda_2$ balance the terms.
After adaptation, a small reasoning stage fine-tunes on templated multi-step questions over the same charts. This lets us measure how much reasoning data is still needed once perception is repaired.
Baselines
All trained baselines below start from the same LLaVA-1.5-7B checkpoint. They receive the same total GPU-hours for training, the same pool of rendered chart images where the method allows it, and the same hyperparameter search budget (an equal number of configurations on a held-out synthetic validation split, never on test benchmarks).
The do-nothing reference is the unmodified LLaVA-1.5-7B. For orientation, MathVista reports open-source models such as LLaVA hovering between 20% and 26%, against a frequent-guess baseline of 26.3%. Plot Twist reports collective accuracy on elementary chart questions often below 25% across the models it tests.
The strongest data-side competitor is fine-tuning on the released Multimodal Self-Instruct training set of 62,476 instructions. It is the most direct prior use of code-rendered abstract images on this exact backbone. The paper reports a 19.8% improvement in chart accuracy for LLaVA-1.5-7B. We will rerun it under our compute budget on our evaluation suite. Beating it means higher accuracy on Chart101 and human-written ChartQA with no more trained examples and substantially less generation compute.
The strongest chart-specialist competitor is Synthesize Step-by-Step. It reports 53.9% on human-written ChartQA and 72.6% relaxed accuracy on ChartQA overall with augmented MatCha. Its backbone differs, so we treat these numbers as an external reference rather than a matched comparison. We will report whether our 7B model reaches that human-split number. The matched version of this competitor is our own ablation in which the same charts are used with Python-rationale supervision through a frozen encoder.
The test-time competitor is BeaconChart, which reports 91.60% on ChartQA and 72.05% on ChartX with Qwen3-VL-32B. It is a larger-model, multi-agent ceiling. We do not expect to match it at 7B. Instead we compare accuracy per unit of inference compute, and we also apply our adapted encoder inside a reduced plan–extract–verify loop to see whether the gains add up.
The key ablations isolate the proposal's ingredients. The first is end-to-end unfreezing with only $\mathcal{L}_{\text{LM}}$ on the identical rendered charts: no readout term and no pairs. This tests whether encoder supervision matters or simply letting the encoder move is enough. The second drops the pair term, keeping the readout term with independent renderings. The third replaces probe-guided allocation with uniform sampling. The fourth supervises with a whole linearized-table target in the style of chart derendering instead of element-level minimal pairs.
Why not just…?
Why not just unfreeze the encoder and fine-tune end-to-end on existing synthetic chart instructions? This is the most serious threat. It may be that letting gradients from the answer loss reach the encoder already repairs perception, and that the pairs and readout term add nothing. Nothing in the file reports this experiment for Multimodal Self-Instruct or Synthesize Step-by-Step, which both keep perception fixed. Our first ablation runs exactly this comparison on the same images and compute. If it matches the full method on Chart101 and on the misleading-title split, the minimal-pair idea is unnecessary, and we will report it that way. The remaining contribution would then be the probe and the zero-rejection generator as a cost result.
Why not just use a higher-resolution encoder? Multimodal Self-Instruct names 336×336 resolution as a bottleneck, so resolution is a plausible cause of some failures. It does not explain the title shortcut, the swings between Matplotlib and JavaScript renderings, or the gridline drop reported in Plot Twist, because those images are not harder to resolve. We will include a higher-resolution-crop baseline and the combination with our method. If resolution alone closes the style-flip and title-flip gaps, our hypothesis about learned invariances is wrong.
Why not just convert charts to tables with DePLOT or OCR and give them to a strong text model? ChartQA reports that table-based pipelines fail on questions about visual attributes such as "the orange line," reaching 45.52% overall and about 30% on human-written questions. Synthesize Step-by-Step inherits every DePLOT extraction error. GeoSQA shows that a perfect transcription still leaves reasoning failures. We include a DePLOT-plus-LLM pipeline baseline, and a matched derendering-target ablation to check whether element-level contrast beats whole-table supervision.
Why not just verify at test time, or prompt with chain-of-thought? BeaconChart works, but it requires several 32B-model calls per question and fails when its planner picks the wrong fields. MATH-Vision finds that chain-of-thought prompting sometimes lowers multimodal accuracy because it compounds early perceptual mistakes. DCoT depends on the model locating the right region in its first turn. We report a DCoT baseline and BeaconChart-style verification on top of both the frozen and the adapted encoder. If verification helps the frozen encoder but adds little to the adapted one, that is direct evidence that our method removes the errors verification exists to catch.
Proposed Experiments
Evaluation is split into perception, reasoning, transfer and cost.
Perception is measured with three instruments. The first is Plot Twist's Chart101, reporting per-question and collective accuracy, plus its gridline, marker and title perturbations. The second is a held-out minimal-pair test set rendered with a library withheld from training. The third is the linear probe's accuracy on encoder features.
Reasoning is measured on human-written and machine-generated ChartQA and on ChartX.
Transfer beyond charts is measured on MathVista, with emphasis on its FunctionQA and plot-based subsets; on MATH-Vision; and on the released VeriSciQA data as an evaluation of real scientific figures.
Cost is reported as GPU-hours, frontier-API calls, and the yield of usable examples per generated candidate. This can be set against VeriSciQA's reported 3.0% pass rate. No reported result was found on Lacuna for the generation GPU-hours of Multimodal Self-Instruct or Synthesize Step-by-Step, so we will measure their generation cost by re-running small slices of each released pipeline.
Three checks decide the outcome.
First, the downstream gain must be mediated by perception. Across all training variants, improvement on human-written ChartQA should track improvement in probe accuracy. A run that raises probe accuracy without raising ChartQA, or the reverse, falsifies the causal story.
Second, the shortcut gap must shrink. The difference between informative-title and misleading-title accuracy on Chart101-style charts should narrow substantially relative to the Multimodal Self-Instruct baseline. If it does not, the model is still reading text rather than marks.
Third, the error mix must shift. On a manually categorized sample of MATH-Vision and ChartQA failures, using that benchmark's vision-versus-reasoning error taxonomy, the share of vision recognition errors should fall. The share of reasoning errors should then rise, since the reasoning failures that GeoSQA and MATH-Vision document should remain and become the dominant cause.
The method succeeds if it beats the matched end-to-end-unfreezing ablation and the Multimodal Self-Instruct baseline on perception and human-written ChartQA, while using no frontier-model calls and keeping nearly every generated example. If end-to-end unfreezing ties it, we report that plainly: the useful contribution narrows to the measurement tool and the low-cost generator.
References
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model
- GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- BeaconChart: Verifier-Guided Corrective Instruction for Chart Understanding
- DCoT: Dual Chain-of-Thought Prompting for Large Multimodal Models
- VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering
- InstructDET: Diversifying Referring Object Detection with Generalized Instructions
- DISCO: Distilling Counterfactuals with Large Language Models
- ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts