Vision-language models (systems that read an image and answer questions about it in text) often answer correctly when a chart's numbers are typed out as a table, and then fail on the same question when they have to read those numbers off the chart. Small changes that leave the data untouched also shift their answers. Adding grid lines dropped one model's accuracy on elementary chart questions from 29.5% to 21.3%, and replacing a chart's title with misleading text pulled accuracy from as high as 79% down to the 30% range (Plot Twist). Graph learning has a standard remedy for this kind of fragility. Two views of the same object pass through one encoder with shared weights, and a consistency loss forces the two views to produce the same representation. The model then has to ignore whatever differs between the views. We propose carrying that recipe into a chart-reading model, with one change in which view acts as the reference. The image branch should not be made to agree with another rendering of the same chart. It should be made to agree with the chart's source data table, processed by the same model as plain text.
Concretely, the same model is trained on two paired inputs. One is a chart image drawn from a data table, using a randomly varied style: grid lines, marker shapes, palettes, plotting library, and stripped or misleading titles. The other is that table written as text. In both, the question and the reasoning tokens are identical. We penalize differences between the model's internal states at those shared tokens, so that reasoning over the image is driven into the same internal state as reasoning over the clean table. At inference time only the image is given. The claim is that this aims at two problems at once: the image-versus-text reasoning gap, and sensitivity to rendering style. Ordinary rendering augmentation and answer-level distillation do not do this.
Core Claim
Let $x^{\text{img}}_r$ be a chart image drawn from table $T$ with rendering style $r$, let $x^{\text{tab}}$ be $T$ serialized as text, and let $q$ be a question with gold rationale and answer $y$. One model with parameters $\theta$ reads $(x^{\text{img}}_r, q, y)$ and $(x^{\text{tab}}, q, y)$. Write $h^{\text{img}}_{\ell,t}$ and $h^{\text{tab}}_{\ell,t}$ for its hidden states at layer $\ell$ and token position $t$, restricted to the question and rationale tokens, which are the same in both branches. The training loss is
$$ \mathcal{L}(\theta) = \mathrm{CE}(y \mid x^{\text{img}}_r, q) + \mathrm{CE}(y \mid x^{\text{tab}}, q) + \lambda \sum_{\ell \in \mathcal{S}} \sum_{t} \big\| h^{\text{img}}_{\ell,t} - \mathrm{sg}(h^{\text{tab}}_{\ell,t}) \big\|^2 . $$
Here CE is next-token cross-entropy, $\mathcal{S}$ is a chosen set of middle-to-late layers, $\lambda$ sets the strength of the consistency term, and $\mathrm{sg}$ is stop-gradient. The stop-gradient makes the text branch an anchor that the image branch moves toward. The text branch is never pulled toward the image branch's errors.
The table view is the right reference for a specific reason. Two renderings of one chart can share a shortcut, for example a title that gives the answer away, so making them agree does not remove the shortcut. The table contains no title cue and no rendering style. Agreeing with it therefore defines invariance with respect to the data itself.
We predict two things. First, compared with augmentation alone at matched compute, the method will shrink the image-versus-table accuracy gap. Second, it will cut answer flips under rendering changes that were held out of training, and this robustness will not come at a cost in clean accuracy.
Relation to Prior Work
The method's form comes from graph contrastive learning. NoiseHGNN pairs a noisy heterogeneous graph with a cleaner, similarity-derived graph inside a shared-parameter dual-branch encoder, so that the cleaner view guides which noisy links to ignore. Our table branch plays the role of its similarity graph: a cleaner view of the same instance that acts as a guide. Biased Edge Dropout in NIFTY uses a weight-shared Siamese network to make representations invariant to a specific spurious structure, demographic homophily. Our misleading-title renders target a specific spurious cue in the same targeted way. CI-GCL warns that naive random augmentation can destroy the very structure that carries meaning. We take that warning as a design constraint: every perturbation must leave the plotted data exactly as it is.
Three findings motivate the work. Plot Twist documents the rendering and title fragility. MATH-Vision finds that reasoning errors (42.2% of GPT-4V failures) outnumber recognition errors. TIGER shows that models fail on images of text problems they solve as text. It trains with rewards on rendered math text, and it explicitly leaves charts and diagrams open. We differ from TIGER in two ways. We work on charts, where quantities are encoded visually rather than as rendered characters. And we use a dense, per-token internal-state signal instead of a reward on the final answer.
Several systems connect the two directions, and they also mark out where we must differ. RAPID converts the image into text and passes that text to a frozen reasoner. The position paper Reasoning After Perception Means Reasoning Without Vision argues that collapsing the image to text throws away spatial structure. Our method uses text only during training, as a target. At inference the model still attends to the pixels, so it sits on the "reason within perception" side of that argument. Multimodal Graph Contrastive Learning for ChartQA and MSG-Chart align separate visual and OCR graphs built by external detectors, and both papers report that detector and OCR errors propagate unchecked. We use no detector, and our two branches share all weights. From Seeing to Thinking separates perception and reasoning into training stages but adds no consistency between views.
Research Program
We will build a rendering engine that draws bar, line, scatter and pie charts from tables using Matplotlib and a JavaScript plotting library, following the perturbation axes of Plot Twist. The perturbation families are grid lines, marker shapes, palettes, font and layout, title removal, and misleading titles. Two families, the JavaScript engine and palette shifts, are reserved entirely for testing.
Tables will come from PlotQA-style synthetic data and from ChartQA charts where the underlying tables are available. We will check during data preparation whether ChartQA tables are available; the file does not confirm it. Questions and rationales will be generated with templates and programs in the style of Synthesize Step-by-Step, so that every answer can be checked.
The base model is Qwen2.5-VL-7B, the model RAPID uses, fine-tuned with LoRA. The main analysis tracks the per-layer distance between the image and table hidden states on held-out charts. If the mechanism works as claimed, this distance should predict errors on image inputs. That would turn the representation gap into something we can measure, not just a story.
Baselines
The do-nothing reference is the base model with no further training, evaluated on clean and perturbed renders. For this model on ChartQA there is no reported result found on Lacuna, so we will measure it ourselves. For Chart101-style probing, the reference points are PaLI-3 at 29.5% falling to 21.3% with grid lines, and collective accuracy below 25% across models (Plot Twist).
The first ablation, and the most important one, is augmentation-only fine-tuning. It uses the same model, the same rendered images, questions, rationales, token budget and hyperparameter search, but drops both the table branch and the consistency term. If our method cannot beat this, the dual-branch idea adds nothing.
The second ablation is output-level distillation. Here the table branch's generated rationales and answers supervise the image branch, but there is no hidden-state term. This separates "learning from the table" from "matching internal states." A third, cheaper ablation keeps the consistency term but uses a second rendering as the anchor instead of the table. This tests whether the table matters specifically or whether any second view would do.
The strongest external competitors on chart accuracy are derender-then-reason systems. ChartPaLI-5B with program-of-thought reports 81.3% ChartQA accuracy (Chart-based Reasoning). On the harder human-written split, MatCha with synthesized rationales reports 54% (Synthesize Step-by-Step) and VL-T5 with MSG-Chart reports 38% (MSG-Chart). We will reimplement a RAPID-style pipeline at matched size, in which our base model writes a table or caption and Qwen3-8B reasons over it, and train it on the same data. We will also run TIGER-style reward training on our chart pairs, where the reward requires the image answer to match the table answer, using the same number of rollouts' worth of compute.
To count as beating these baselines, our method must do three things relative to augmentation-only and distillation, averaged over three seeds and outside seed variance. It must reduce the held-out flip rate. It must reduce the image-versus-table gap. And it must match or exceed their clean ChartQA accuracy. Against the derender pipeline, the claim is narrower. We need to win on robustness to held-out rendering families and to misleading titles, and on questions about shape or trend that depend on geometry. We do not claim to beat it on headline ChartQA accuracy.
Why not just…?
Why not just augment the training data with many renderings? This is the most serious alternative. Augmentation teaches the model to give correct answers on varied renders, but nothing requires those answers to come from the data rather than from a cue shared across renders. The misleading-title family and the held-out engine and palette families are built to expose that difference. If augmentation-only matches our method on held-out flip rate and on the image-versus-table gap, we will report that the consistency term is unnecessary and drop the claim.
Why not just derender the chart to a table and let a language model reason? This works well on ChartQA (81.3% for ChartPaLI-5B with program-of-thought). However, the derendering step is itself a perception model, and it is exposed to the same rendering fragility documented in Plot Twist. It also discards geometry, which is the point of the position paper. Our evaluation on held-out renders and geometric questions will show whether the pipeline's accuracy survives these conditions. If the pipeline wins there too, the case for a single robust model weakens to a cost and latency argument, and we will say so.
Why not just distill the table branch's answers, or use TIGER-style rewards? Both are cheaper to explain, and both supervise only outputs. TIGER reports about a 15% gain on MathVerse, but only for images of text (TIGER). The output-distillation and reward baselines described above test whether dense per-token matching of internal states adds anything beyond output supervision. The test is the layer-wise distance analysis: under our method, that distance should shrink and predict errors, and under the output-only methods it should not.
Proposed Experiments
We train on the synthetic paired corpus and evaluate in four places. The first is ChartQA, reported separately for human-written and machine-generated questions. The second is Chart101-style elementary probes regenerated under every perturbation family, scored by accuracy, collective accuracy and flip rate. The third is ChartX, for zero-shot transfer across chart types. The fourth is MathVista (MathVista) and MATH-Vision, to check that general visual math does not get worse.
Three checks decide the outcome. The first is the held-out rendering families. A method that has only memorized the training perturbations will fail there. The second is misleading titles, where the difference in accuracy between the true-title and misleading-title conditions should shrink under our method more than under augmentation. The third is the image-versus-table gap on matched questions, together with whether the internal-state distance predicts errors.
The idea is falsified if augmentation-only matches the full method on all three checks.
References
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- NoiseHGNN: Synthesized Similarity Graph-Based Neural Network for Noised Heterogeneous Graph Representation Learning
- Biased Edge Dropout in NIFTY for Fair Graph Representation Learning
- Community-Invariant Graph Contrastive Learning
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- TIGER: Bridging the Multimodal Reasoning-Access Gap via Modality Counterfactuals
- Reasoning-Aligned Perception Decoupling for Scalable Multi-modal Reasoning
- Position: Reasoning After Perception Means Reasoning Without Vision
- Graph-Based Multimodal Contrastive Learning for Chart Question Answering
- MSG-Chart: Multimodal Scene Graph for ChartQA
- From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs
- MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts