Checking Chart-Reading Programs One Data Point at a Time

A promising way to get multimodal models to reason about charts is to have them "derender" the image. The model writes the Python plotting code and data table that would have produced the chart, then answers the question by running a query over that table. RECODE does this and checks each reconstruction by re-rendering the code and measuring pixel-level mean squared error against the original image. The weakness is the check itself. Pixel error is dominated by fonts, padding and line thickness, so it penalizes harmless style differences. It is also nearly blind to the errors that change answers: two series with their colors swapped, one bar shifted by a single tick, a value copied from a misleading title. These are silent errors. The program runs, the picture looks right, and the answer is wrong.

We propose to replace the whole-image pixel check with a per-datum grounding test. For every value in the reconstructed table, we render only that value's mark, record exactly which pixels and which color it should occupy, and ask whether the original image has matching ink there. Because this test is deterministic and needs no model, it cannot share the model's biases. We then trace the query program's execution to find which table cells the final answer actually depends on. Only those cells are verified and re-read. The result is an oracle-free detector that says which data point is wrong and whether that error matters for the answer. Contextual noise is the setting where we expect it to earn its keep: misleading titles, narrative captions, grid lines, and changes of marker style or plotting library.

Core Claim

In program-aided chart reasoning, silent errors are best caught by two things together. The first is a check done in data space, one mark per value, rather than in pixel space. The second is restricting that check to the cells the answer depends on. We claim this combination detects wrong answers better than RECODE's pixel critic, better than majority voting over repeated derenderings, and better than asking a language model to judge its own code. We also claim the advantage grows under contextual and visual noise.

The reason to expect this comes from three lines of evidence. Plot Twist shows that models read answers off chart titles instead of the marks. With misleading titles, accuracy falls from as high as 79% to the 30% range, and adding grid lines drops PaLI-3 from 29.5% to 21.3%. Less is More shows that narrative text makes code generators latch onto irrelevant nouns. A derenderer exposed to either kind of noise should write tables that agree with the text rather than the ink. A test that looks only at ink, at the positions the table claims, is immune to that pull.

Two further papers explain why the check must not be the model itself. The Counterfeit Conundrum shows that code models, asked to predict what buggy code does, predict what the correct code would have done. Most non-GPT-4 models reach only about 60% accuracy at correctness checking. Revisit Self-Debugging shows that model-written tests inherit the code's bugs, and that inspecting intermediate program state works where pass/fail feedback does not.

Relation to Prior Work

Our closest neighbor is RECODE. We keep its derender-then-query structure and replace its critic. RECODE compares whole re-rendered images and never inspects the data table or the query's dependence on it. We compare individual marks and use the query's execution trace.

Synthesize Step-by-Step trains a model to break chart questions into executable Python steps. It does not verify that the values those steps consume are grounded in the image. Evaluating Intermediate Reasoning of Code-Assisted LLMs motivates our focus: correct final answers often hide unsound code in text-only math. We carry that concern to visual inputs, where the unsound part is usually the data rather than the logic.

We borrow the idea of confirming a diagnosis by a targeted replay from REFLECT. REFLECT patches the suspected error, reruns the trace, and treats a changed outcome as causal evidence. It needs ground-truth answers to confirm that the outcome flipped. We get a test-time substitute from the image itself: a flagged cell is confirmed when re-reading it changes the answer and the new value passes the mark test.

DCoT supplies the repair tactic of cropping to a region so the model focuses on it. DCoT picks that crop using the model's own first guess about where to look. Our crops are placed by rendering the disputed datum, so a distracted first guess cannot misplace them.

Two results constrain the design. MATH-Vision reports that chain-of-thought can hurt multimodal models and that reasoning errors outnumber recognition errors (42.2% vs 31.9% for GPT-4V). This tells us not to expect the method to help on geometry-heavy reasoning. We therefore scope it to charts, where the dominant silent error is misreading the data. Plot Twist's finding that models fail elementary probes is a second constraint. It means we should not use the model to answer per-cell probe questions about the image, which is why our core test uses no model at all.

Research Program

The pipeline has four stages. First, a multimodal model derenders the chart into a data table plus plotting code, following RECODE's prompting. Second, we calibrate the axes. Tick labels are read by optical character recognition and cross-checked against the code's axis limits, and a mismatch there is flagged as an error of its own.

Third, for each cell, we render that datum alone in the reconstructed style to obtain its pixel footprint and color. We score agreement with the original image only inside that footprint, using color distance on ink pixels. We also run a symmetric check in the other direction: ink of each series' color in the original image that no reconstructed datum accounts for is flagged as a missing point.

Fourth, the query program runs under a tracer that records which table cells flow into the returned value. This extends the intermediate-state tracing of Revisit Self-Debugging from code variables to table cells. A cell that fails the mark test and lies on the answer's dependency path triggers repair. Repair means re-reading only that region from a crop, with the title and caption masked, and re-running the query. If no answer-relevant cell fails, the answer is accepted. The per-cell failure scores, combined over answer-relevant cells, serve as a confidence score for selective answering.

We will study which error types the test sees and which it misses. We expect it to see value, series-swap, missing-point and axis-calibration errors. We expect it to miss errors in the query logic itself, and occluded or overlapping marks.

Evaluation Suite

We evaluate on the human-written split of ChartQA, on CharXiv-Reasoning, and on Chart101 from Plot Twist. We also build a noise suite from ChartQA and Chart101 images. For charts whose underlying data we have or can recover, we re-render them with the Plot Twist perturbations: grid lines, different marker styles, a different plotting library, and titles that state a wrong answer. For every chart, we also embed it in long narrative captions modeled on the story-style padding studied in Less is More.

Every perturbation leaves the answer unchanged, so any change in accuracy is caused by the noise. Because we rendered the perturbed charts ourselves, we also know the true table for each one. That lets us score cell-level localization directly, not just final answers.

Baselines

The naive reference point is the same multimodal backbone answering directly from the image, with and without chain-of-thought. This measures the gap derendering closes.

The strongest existing method is RECODE with its pixel critic and iterative refinement. It reports 93.2% accuracy on ChartQA and 77% on CharXiv-Reasoning, a 19-point absolute gain over Gemini 2.5 Pro, which puts that baseline at about 58%. We will reimplement RECODE on our backbone, using the same derendering prompts, the same number of refinement rounds, and the same total model-call budget as our method. We first check that we approximately reproduce the reported ChartQA number. On the noise suite, we found no reported result on Lacuna for RECODE under grid, marker, library, misleading-title or narrative perturbations. Those numbers will be ours.

As a second strong competitor on the human split, we include the Synthesize Step-by-Step reasoning approach, reported at 53.9% on ChartQA human-written.

We also run a sampling baseline: draw k derenderings at temperature and take the majority answer, with k chosen so its compute matches our pipeline. We include it because The Counterfeit Conundrum finds that resampling often beats repair.

Finally, an ablation isolates each key ingredient. One variant runs the mark test on all cells without the dependency trace. Another uses the dependency trace with RECODE's pixel error restricted to answer-relevant regions instead of the mark test.

Beating RECODE means higher accuracy on ChartQA and CharXiv-Reasoning at matched compute, a clearly smaller drop from clean to perturbed charts, and a better separation of right from wrong answers by the confidence score. We measure that separation as area under the ROC curve, and also as accuracy at a fixed rate of answers withheld.

Why not just…?

Why not just resample the derendering several times and vote? This is the real threat. The Counterfeit Conundrum's finding that fresh samples beat patching points in its favor. Our argument is that errors caused by noise are shared across samples. If the title says 40, most samples will write 40. Voting then amplifies the error, while the mark test checks ink that the title cannot move. The compute-matched voting baseline settles this on the misleading-title and narrative splits. If voting matches us there, the data-space critic adds nothing and we would recast the contribution as the dependency-restricted confidence score alone.

Why not just use a better pixel metric, such as error computed only over data ink? Masking non-data pixels fixes the oversensitivity to fonts and padding. It still returns one number per image, so it cannot say which cell is wrong or whether that cell matters. The ablation that restricts pixel error to answer-relevant regions is exactly this alternative. If it performs as well, the per-datum rendering is unnecessary, and we would report that.

Why not just ask the model to review its code, or translate it back to English and compare with the question? Back-translation reranking helps on SQL: von Däniken et al. lift Spider accuracy by 2.59 points. But it checks whether the query matches the question. It does not check whether the data matches the image. Self-review runs into the intention bias documented in The Counterfeit Conundrum and the test bias documented in Revisit Self-Debugging. We include a model-as-judge detector as a baseline for the separation metric.

Why not just strip titles and captions before derendering? For title and caption noise this is cheap and may work well, so it is included as a preprocessing baseline for every method. It does nothing for grid lines, marker changes, library changes or genuine misreads. Less is More also shows that removing noise leaves reasoning errors in place, with only about a 2% gain on competition-level problems. It also discards titles that carry legitimate information, such as units.

Decisive Checks

The first decisive check is on the synthetic perturbation suite, where true tables are known. The mark test should flag the corrupted cell in most silent-error cases where RECODE's pixel critic ranks the wrong reconstruction as acceptable. If the pixel critic and the mark test flag the same cases, the premise fails.

The second is the noise gap. Across the Plot Twist perturbations, our method's accuracy drop from clean to perturbed should be substantially smaller than RECODE's and voting's under matched compute.

The third is selective answering on CharXiv-Reasoning, where no true tables exist. The dependency-restricted confidence score should beat both the voting-agreement score and the model-as-judge score at identifying wrong answers.

A negative outcome on the first check alone would falsify the core claim. A negative outcome only on the third would limit the contribution to clean synthetic charts.