Render-and-Compare Verification for Chart Reasoning

Vision-language models can answer hard questions about charts and still misread the basic numbers in them. On the Chart101 probe, top models answer every elementary question about a single chart (axis limits, extremes, individual values) correctly less than 25% of the time. Their scores on harder benchmarks look better largely because they read answers off chart titles. A common and effective fix for weak arithmetic is to have the model convert the chart into a table and then run a short Python program on that table. That fix has a weakness of its own. If the table is wrong, the program computes the wrong answer with full confidence, and nothing in the pipeline checks whether the table matches the picture.

We propose to check the table by drawing it. The model converts the chart into a table plus a small declarative plotting specification (chart type, colors, axis ranges). We render that specification back into an image and compare the rendering to the original, mark by mark, in pixel space. Marks that disagree point to specific cells that were probably misread. Those cells are re-read from image crops before any arithmetic runs. The same disagreement score works as a label-free reward for training the chart-to-table step on unlabeled real-world charts. Our bet is that comparing image to image avoids the hardest part of the cross-modal problem. A table-to-image check does not need text and pixels to line up in a shared embedding space, because both sides of the comparison are pictures.

Core Claim

Several training-time methods try to reduce the cross-modal gap between structured data and visual tokens. ChartMoE aligns connector experts to CSV, JSON and code views of a chart. TURBO uses structured tables as training-only privileged information. CoReTab pretrains table recognition against table code. None of them can tell, at test time, whether a particular extracted cell is faithful to the image. Our claim has two parts. First, a rendering of the model's own structured hypothesis, compared locally against the input image, gives a per-cell error signal that predicts extraction mistakes well enough to act on. Second, fixing flagged cells before symbolic execution closes much of the gap between complex-question accuracy and elementary-read accuracy documented by Plot Twist. It should also remove the dependence on titles. We treat the three directions unequally. The structured-reasoning gap supplies the reasoning back end, which is execute-don't-generate, following PoTable and TART. The multimodal reasoning gap supplies the failure we target, which is cascading perception errors. The cross-modal representation gap is the lens: we avoid it rather than learn to close it.

Relation to Prior Work

The closest pipelines already combine chart-to-table conversion with programs, so the proposal must differ in the verification step.

Chart-based Reasoning trains ChartPaLI-5B on chart-to-table translation and synthetic rationales, then optionally adds Program-of-Thought execution. Synthesize Step-by-Step builds Python rationales on top of DePlot's chart-to-table conversion. That paper names the core limitation itself: derendering errors propagate through every rationale built on them. ChartAssistant documents the same cascade for sequential chain-of-thought. All three trust the extracted table. We keep their extract-then-execute structure and add a check on the extraction.

Chart-to-code generation already exists, but only as a test. ChartMimic evaluates how faithfully models convert charts into Matplotlib code and finds that small open models produce executable code less than half the time. That finding constrains our design. We do not ask the model for free-form plotting code. We ask for a fixed declarative specification that a deterministic template renderer always executes. We also use the rendering as a verifier inside question answering, which ChartMimic does not do.

Representation-level approaches play a different role. ChartMoE, MSG-Chart (visual and textual scene graphs as extra features) and the embedding-alignment line (such as ReViSE with distribution matching, and Ex-MCR showing single projectors cannot close both intra-modal and cross-modal gaps) all improve the internal representation. Ex-MCR's negative result motivates us to move the comparison into a single modality instead.

Two papers point the same way for our reward. TURBO and CoReTab both use reinforcement learning with correctness rewards, which require labeled answers. Our render-consistency reward requires only chart images.

Research Program

Inference loop. A base vision-language model, Qwen2.5-VL-7B (chosen to match CoReTab's backbone), emits three things for an input chart:

  • a table;
  • a declarative specification covering mark type, per-series color, axis ranges and tick positions, legend entries and title;
  • for each cell, the model's claimed pixel location of the corresponding mark.

A template renderer then draws the specification at the input resolution. The comparison is local, not global. For each cell we take the rendered mark's region and the corresponding region of the input. We compute a style-tolerant discrepancy: the offset between color-masked mark extents along the value axis, measured after the rendered and input axes are aligned by their tick positions. Cells above a threshold are cropped at higher resolution and re-read by the model with the crop and the axis ticks in context. The loop repeats up to $K$ times. A pandas program then answers the question from the corrected table. The specification carries colors and legend bindings, so color-grounded questions become column selections rather than guesses. ChartPaLI's authors report that color-grounded questions are exactly where their table-only rationales fail.

Training signal. The per-cell discrepancies are summed into a scalar render-consistency score $D$. We use $r = -D$ as a reward for policy optimization of the extraction step, alongside a small format reward. Training uses unlabeled real-world chart images, such as RealCQA images and the chart images in ChartSFT, with their labels withheld. The comparison arm is the same optimization using answer-correctness rewards on labeled ChartQA training data. This isolates whether label-free perceptual feedback adds anything beyond answer supervision.

Analysis. We will measure how well $D$ predicts true cell error. Synthetic charts with known underlying tables (PlotQA, DVQA, the machine-generated ChartQA split) provide ground truth. This is the load-bearing empirical question: if the discrepancy does not track error, the idea fails at its root. We also report where the method cannot help:

  • charts with stylized or three-dimensional rendering that the template renderer cannot reproduce;
  • tables with embedded icons from MMTBench.

The loop abstains on such inputs, and we report its behavior there separately rather than hiding it. Table images (MMTab, AIT-QA), where HTML can be rendered back with a browser and compared cell by cell, are a secondary domain once charts work.

Baselines

Naive reference. The same Qwen2.5-VL-7B answers directly with chain-of-thought, under our prompts and decoding. This measures the gap we start from. On multimodal math, MATH-Vision reports that chain-of-thought sometimes lowers accuracy, so we also report direct answering.

Strongest end-to-end competitor. ChartMoE reports 84.64% on ChartQA, against a previous best of 80.48%. We compare against its published number. Where released weights allow, we also rerun it on Chart101 and on the perturbed ChartQA splits described below. The perturbed splits are where we expect representation-only methods to be exposed.

Strongest extract-then-execute competitor. ChartPaLI-5B with Program-of-Thought reports 81.3% on ChartQA. The human-written split is the hard one. There, Synthesize Step-by-Step reports an improvement from 38.2% to 53.9%, and ChartAssistant's ChartAst-S reports 65.9%. We re-implement the ChartPaLI-style pipeline on our own backbone: extract a table, run a program, no verification. This controls for backbone differences. It is the ablation that isolates render-and-compare, because the pipelines are identical except for the verification loop.

Compute-matched self-consistency. We sample $N$ tables and take a per-cell median or majority. $N$ is chosen so that total model forward passes match our loop's average, counting re-reads.

Model self-critique. The model is shown its own table beside the image and asked to fix errors, with the same number of rounds.

Global versus local comparison. A global-similarity variant compares whole images by embedding distance instead of per-mark discrepancy. It tests whether locality, rather than rendering itself, carries the gain.

Training ablations. The label-free reward is compared against the answer-correctness reward and against supervised chart-to-table training on the same images at the same compute.

What counts as winning. For the loop, beating the unverified extract-then-execute pipeline and the compute-matched self-consistency baseline on the ChartQA human split and on Chart101 collective accuracy, with non-overlapping bootstrap intervals. For the training reward, matching or exceeding the answer-reward arm without using answer labels. For published comparisons, we treat ChartMoE's 84.64% as the bar on standard ChartQA. We do not claim a win there unless we exceed it. We expect the clearer wins on perturbed and elementary-read evaluations.

Why not just…?

Why not just scale the model, or train a better converter? Scale does not fix elementary reads. Plot Twist's collective accuracy below 25% covers GPT-4V and Gemini Pro Vision. Better converter training is the more serious version of this objection: ChartAssistant's chart-to-table stage reports a 29.5% improvement over UniChart on RealCQA. Our training ablation answers it directly. It gives supervised chart-to-table training the same images and compute as our reward. If supervised training matches the render reward, the training contribution collapses to the inference loop alone, and we will say so.

Why not just sample several tables and vote? This is the real threat. Independent samples could catch the same misreads without any renderer. The difference we expect is correlation. A model that reads a title or trusts a misleading gridline makes the same mistake in every sample, while a rendered comparison measures against the pixels. The compute-matched self-consistency baseline settles this, especially on Plot Twist's title-removal and gridline perturbations. If voting closes most of the gap there, we would reframe the renderer as an error localizer that makes voting cheaper rather than as a better verifier.

Why not just ask the model to check its own table? Triples as the Key reports that language models asked to verify tend to justify their earlier errors. The Plot Twist results suggest the same vision failure would recur when the model looks again. The self-critique baseline measures this directly.

Why not just run programs on the extracted table, as ChartPaLI and PoTable do? We keep execution, since programs remove arithmetic errors. Execution cannot remove perception errors, and ChartAssistant and Synthesize Step-by-Step both identify those errors as the cascade source. The unverified-pipeline ablation measures exactly this difference.

Proposed Experiments and Decisive Checks

The first check is diagnostic. On PlotQA and the machine-generated ChartQA split, where underlying tables are known, we measure how well per-cell discrepancy separates correct from incorrect cells (area under the ROC curve). We also measure what fraction of wrong final answers trace back to at least one flagged cell. If the area under the curve is near chance, we stop.

The second check is robustness. We rebuild Plot Twist's perturbations on Chart101 and on the ChartQA human split: titles removed, titles swapped with misleading ones, and gridlines added. The reported vulnerabilities give the baseline pattern. Title-containing charts reached up to 79% accuracy against roughly 30% without titles, and gridlines dropped PaLI-3 from 29.5% to 21.3%. Success means our loop's accuracy changes far less under each perturbation than the unverified pipeline's and the self-consistency baseline's. It also means Chart101 collective accuracy rises clearly above the reported sub-25% level. For reference, the MatCha model fine-tuned on Synthesize Step-by-Step's rationales reports 92.89% on PlotQA, leaving limited headroom there, so PlotQA serves the diagnostic rather than the headline result.

The third check is standard accuracy: ChartQA overall and human split, ChartBench (where ChartMoE reports 51.67%), and a color-grounded subset of ChartQA questions that we tag by keyword. The fourth check is the label-free reward: train on unlabeled RealCQA and ChartSFT images, then evaluate on held-out ChartQA human questions against the answer-reward and supervised arms. We report latency throughout, because TART and PoTable show that iterative pipelines are costly. We also report the abstention rate on charts the renderer cannot reproduce, so the method's coverage is visible alongside its accuracy.