When a language model writes plotting code, the program usually runs without errors and still draws the wrong picture. A data series can vanish behind another, a label can be clipped, an axis can be truncated, or two categories can get the same color. The interpreter returns success, so the usual debugging loop gets no error signal. Current fixes take one of two routes. They compare the rendered image against a gold target chart, which rarely exists in practice. Or they ask a vision-language model (a model that reads images and writes text) to critique the picture in prose. That prose critique passes the feedback back through the lossy image-to-text step that caused much of the trouble.
We propose a third source of evidence that is already present and exact. After the code runs, the plotting library holds a complete in-memory object for the figure. It records every drawn element, the numbers it encodes, its color, its pixel bounding box, its stacking order, and the line of code that created it. The user's own data table and request act as the gold standard. We cross-examine three things against each other: the source data and request, this live figure object, and the rendered pixels. Each mismatch is tied to a specific line of code, and the model is asked to repair only that line. The bet is that exact, line-localized, reference-free discrepancies get past the two-to-four-iteration plateau of refinement loops, and do so more reliably than either a holistic image critic or a gold-image diff.
Core Claim
In data-to-chart generation, reference-free refinement does not need to go through a vision model's prose description of the output. Most chart defects fall into two kinds. The first is data-fidelity errors: values, series, or categories that are missing, invented, or misscaled. These can be checked exactly by comparing the source table with the data arrays stored in the executed figure object. The second is perceptual errors: occlusion, overlapping or clipped text, and indistinguishable colors. Most of these can be checked geometrically from the bounding boxes and colors in that same object. A vision model is needed only for a narrow, final question: is the element the figure object says is there actually perceivable in the pixels? Because each element in the figure object can be traced to the line that created it, every discrepancy comes with a location. Localized repair should cause fewer regressions than rewriting the whole program, and should keep improving at iterations where current loops stall.
Positioning Against Prior Work
Three papers motivate the gap. ChartCoder argues that code is a lossless way to represent a chart and trains a code-specialized model with staged "snippet-of-thought" planning. It is single-pass and never looks at what its code produced. MSRL shows the opposite side: text- and code-level rewards are blind to rendered layout defects, and only a visual reward on rendered images moves past the supervised-training plateau. Its visual reward compares against a gold target image, and it trains offline instead of repairing at test time. We reconcile the two. The code's executed state, not the source code text, is close to lossless about what was drawn, and it is also exact about geometry, which is what MSRL's visual judge was being used to recover.
Two lines of work supply our methods. CodePRM scores reasoning steps using execution feedback and refines only the lowest-scoring step, which prevents over-correction. It depends on unit tests, and we replace those with figure-object checks derived from the source data. VDebugger localizes logic errors in visual programs from line-by-line variable traces. It cannot see rendered canvases, and our figure-object trace is the rendering analogue of its variable trace. From the text-evaluation side, the Cross-Examination Framework treats source and output as two knowledge bases and checks coverage (did the output keep everything?) and consistency (did it invent anything?) in both directions. It reports reference-free scores that track reference-based ones (r = 0.846 for consistency). We carry this bidirectional design over to a source table versus an executed figure. Where CEF uses LLM-generated questions, our checks are mostly programmatic, which also avoids its roughly 6–7x inference cost.
The closest competitors are reference-free critics. C² derives query-specific yes/no criteria grounded in visualization design principles. It emits code-edit actions (retain, edit, discard, add) from a judgment of the chart image, and it stops working when the code crashes. Vision-Guided Iterative Refinement for Frontend Code Generation runs a vision critic on rendered webpages and then has a separate code critic translate that prose into edits. That second step is exactly where the model has to guess which line caused a visual defect. ChartIR iterates on multi-aspect differences between rendered and target charts. It needs the target, and it communicates differences in prose. None of these inspects the executed figure object, and none ties a defect to a line.
Two results constrain the design. Plot2Code finds that piling detailed instructions into the prompt can lower executability. Our feedback is therefore deliberately small: one discrepancy, one line, one expected value per repair call. Refinement plateaus (Yan et al.) and regression from unchecked edits (Fang et al.) set our success criterion: improvement must continue past iteration three, and no repair may be accepted if it raises the discrepancy count.
Research Program
The first component is an instrumented executor for Matplotlib, which is the library used by ChartMimic, Plot2Code and CHARTUIE-8K. It runs the candidate script and, before the figure closes, walks the figure's element tree. For each element it records its type, its data arrays in data coordinates, the axis transforms, its pixel bounding box after layout, its face and edge colors, its stacking order, its visibility, and its text content. A lightweight wrapper on the library's element constructors records the creating source line for each element. If the script crashes, the executor keeps the partial figure along with the traceback. This lets the loop repair what was drawn before the crash, which addresses the failure case C² leaves open.
The second component is a cross-examiner with three passes. The source-to-figure pass checks coverage. Every series, category, and value in the user's table, along with every explicit requirement the language model parses from the query (chart type, sorting, log scale, highlighted item), must map to a figure element within tolerance. The 5% data tolerance follows MSRL's data-accuracy reward. The figure-to-source pass checks consistency: no plotted values or labels may lack a source, and axis limits must not exclude source values. The geometric pass checks perceptibility from bounding boxes and colors: text-to-text and text-to-marker overlap, elements clipped by the canvas, data hidden behind higher layers, and series colors closer than a perceptual threshold. Only items the geometric pass cannot settle, such as dense scatter occlusion or legend readability, go to a vision model. Even then it gets a closed question about a single cropped region ("is the series labeled X visible here?"), following the binary-question finding of C². It is never asked an open-ended critique.
The third component is localized repair with rollback. Discrepancies are ranked by severity: data errors first, then requirement violations, then perceptual issues. The repair model receives the source lines tied to the top discrepancy, the expected and observed values, and the surrounding code. It is asked to edit only those lines, in the spirit of CodePRM's step-level refinement. The edited script is re-executed and re-examined. The edit is kept only if the weighted discrepancy count strictly drops; otherwise the loop rolls back and tries the next discrepancy. The loop stops when no discrepancies remain or the call budget runs out.
The fourth, optional component distills accepted repair trajectories into a single-pass generator. CITL reports retaining 24.7% to 50.6% of refinement gains this way. This tests whether the localized signal is also a better training signal than a prose critique. It is secondary to the test-time claim.
Baselines
The naive reference point is single-pass generation from the same backbone with the same prompt and no feedback. Next to it we run unprompted self-refinement: the model sees its rendered chart and is asked to improve it. CITL reports that refining without a critic gains only 1.5% on WebDev Arena, and ChartMimic reports that rendered-output self-reflection helps frontier models. Both observations make this a meaningful lower anchor. We also run best-of-N sampling with the same total number of model calls as our loop, selected by our own discrepancy score. This separates "the score is a good selector" from "localized repair helps."
The strongest reference-free competitor is C²'s CHARTAF-G edit-action feedback, iterated for the same number of rounds, with CHARTAF-S used for candidate selection where applicable. It uses the same backbones (GPT-4o, an InternVL2 model from OpenGVLab/InternVL, and Qwen2-VL), the same call budget, and the same stopping rule. The file gives no numeric CHARTAF-G score on a standard metric; the reported evidence is that 84% of participants preferred CHARTAF-refined charts over initial attempts, and that CHARTAF-G beat nine baselines. We will therefore run it ourselves. A CITL-style pipeline, with a vision critic followed by a code critic and an improver, ported to charts, serves as the second reference-free competitor.
As a reference-using upper reference we run ChartIR, which is allowed to see the held-out target chart. On ChartMimic, ChartIR reports a GPT-4o score of 7.15 with GPT-4o. On Plot2Code, it reports 6.56, up from 5.61 for direct generation. For one-shot strength on ChartMimic, the benchmark paper reports a GPT-4o Customized Mimic score of 83.2 with a 93.2% execution rate. MSRL reports an 83.8% high-level score and 96.5% execution rate after training with target-image rewards. Matching ChartIR without seeing the target would count as a strong result. Beating C² and CITL-for-charts by a statistically clear margin on held-out reference metrics, at equal calls, is the success criterion.
Three ablations isolate the key ingredient. The first is "figure object off": discrepancies come only from a vision model reading the rendered chart back into a table and comparing it with the source. The second is "localization off": the same discrepancy list is given to the model with a whole-program rewrite allowed. The third is "vision check off": figure-object and geometric checks only. The claim requires the first ablation to lose substantially on data fidelity and the second to show more regressions and an earlier plateau.
Why not just…?
Why not just iterate C²'s binary critic longer? C² judges the image, so its data checks inherit the vision model's limits in reading dense numbers. This is the bottleneck ChartCoder documents for charts turned into text. Its edit actions also do not say which line to change. It is still the real threat on perceptual and design-principle criteria, where it may match or beat our geometric checks. We expect our advantage to come mainly from data-fidelity errors and from fewer regressions. The planted-defect study below breaks results down by defect type to show where each method wins.
Why not just have a vision model read the rendered chart back into a table and diff it with the source? This round-trip check is the most plausible simpler alternative, because it is also reference-free and data-grounded. It is exactly our "figure object off" ablation. Its weakness is that pixel-to-number reading is the lossy step itself, and that it localizes nothing. If it comes close to the full method on data fidelity, we would report that the figure object adds little beyond localization.
Why not just sample many candidates and pick the best? Best-of-N with a good scorer is cheap and parallel, and it avoids repair regressions entirely. We include it with our own score as the selector at matched calls. If it matches localized repair, the contribution reduces to the scorer. That would still be a useful finding about reference-free chart evaluation, but it would weaken the repair claim, and we would say so.
Why not just train with target-image rewards as MSRL does? That requires gold targets during training and gives no test-time repair for new data. Our setting assumes no target exists. The distillation experiment tests whether figure-object discrepancies can stand in for MSRL's target-based reward.
Proposed Experiments
The primary benchmark is CHARTUIE-8K, which is reference-free by construction. The other two are ChartMimic's Customized Mimic task and Plot2Code's conditional setting. In both, the new data or instruction is supplied as text and can act as our source, while the reference charts are kept hidden and used only for scoring. We report each benchmark's own metrics. For ChartMimic these are the execution rate, the low-level Text, Layout, Type and Color scores, and the GPT-4o score. For Plot2Code they are the pass rate, the text-match ratio, and the rating. On CHARTUIE-8K we add a human preference study comparing against C².
Three diagnostic checks decide the claim. The first is a planted-defect set. Starting from correct ChartMimic reference scripts, we inject known defects (dropped series, swapped colors, clipped labels, overlapping annotations, wrong axis scale, truncated limits). We then measure detection recall, false-alarm rate, and whether the correct line is localized, comparing figure-object checks, vision read-back, and C²-style critique. The second is a plateau curve: held-out reference score and regression rate at each iteration up to six. The prediction is continued gains past iteration three only with localization plus rollback. The third is a validity check in the style of CEF: correlation between our reference-free discrepancy score and the held-out reference metrics across candidates. A weak correlation would mean the loop is optimizing the wrong thing, whatever its headline gains.
References
- ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation
- Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generation
- CodePRM: Execution Feedback-enhanced Process Reward Model for Code Generation
- VDebugger: Harnessing Execution Feedback for Debugging Visual Programs
- Cross-Examination Framework: A Task-Agnostic Diagnostic for Information Fidelity in Text-to-Text Generation
- C²: Scalable Auto-Feedback for LLM-based Chart Generation
- Vision-Guided Iterative Refinement for Frontend Code Generation
- Improved Iterative Refinement for Chart-to-Code Generation via Structured Instruction
- Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- MEIC: Re-thinking RTL Debug Automation using LLMs