Systems that write plotting code from a description or an image handle bar, line and pie charts reasonably well. They break down on heatmaps, violin plots, grouped layouts and plots nested inside other plots. The most reliable fix today is a loop of cooperating models. One model writes the code, the code is rendered to an image, a vision-language model (a model that reads images and text together) critiques the image, and the code is rewritten. The cycle repeats several times. Such loops work, but each query costs several model calls per round across several rounds, which is a form of ensemble inference overhead that rules them out for interactive notebooks. The obvious remedy is to fine-tune the original code writer on the loop's final outputs. Where this has been measured, it recovers only about a quarter of the loop's improvement.
This note proposes a different way to compress the loop, based on the main lesson of residual networks: learning a small correction on top of an identity mapping is easier than learning the full mapping. We keep the original code writer unchanged and add one trained corrector. The corrector looks at the draft code and its rendered image once, and outputs only an edit (a diff) to the draft. An empty edit means "the draft is fine." The corrector is trained on the edits that a full critic loop made, including edits applied to the corrector's own family of mistakes. The bet has two parts. Turning a near-correct chart into a correct one mostly requires local fixes, such as a missing colorbar, wrong binning, a swapped axis or an off palette. These fixes are learnable in one pass and shared across chart types, even when generating the whole chart is not.
Core Claim
A single render-conditioned corrector that predicts code diffs should recover a much larger share of a multi-agent chart-refinement loop's gain than distilling the loop into the generator. It should do so with two model calls per chart and far fewer generated tokens. The advantage should be largest on rare, complex chart types, because edit patterns transfer across types while whole-chart generation does not. The new element is not refinement or distillation in isolation. It is the shape of the distilled object: a one-step residual corrector whose default output is "no change," trained on the student's own drafts.
Position Relative to Prior Work
Two papers motivate the proposal and set the bar.
AMACE chains three agents:
- a code generator,
- a chart replier that reads the rendered chart,
- a quality evaluator,
plus an early-stopping checker. It reports up to a 29.97% accuracy gain on ChartQA:augmented and up to 69.9% token savings from stopping early. Its limitations are what we target. It evaluates mostly bar and line charts, regenerates full scripts each round, and explicitly leaves distillation into a light single-pass model to future work.
Vision-Guided Iterative Refinement for Frontend Code Generation runs a render-and-critique loop for HTML/CSS, then distills it with LoRA (low-rank adapter fine-tuning) into the base generator. That distillation captured 24.7% of the best loop's improvement and 50.6% of third-cycle gains. The same paper shows that refining without a visual critic gives only 1.5%. We read the 24.7% figure as evidence that asking the first pass to absorb the whole loop is the wrong target. We read the 1.5% figure as a constraint: any corrector must actually condition on the rendered image.
Two papers supply methods we reuse.
- Distilling Knowledge for Search-based Structured Prediction shows that distilling an ensemble on states the student itself explores, not only on reference paths, teaches error recovery to a greedy model (+1.32 LAS on Penn Treebank, +2.65 BLEU on IWSLT De-En). We apply the same idea to drafts. The corrector is trained on teacher-loop repairs of drafts the student generator actually produces. This addresses the mismatch between training and test drafts.
- The residual view comes from the ResNet literature. Characterizing ResNet's Universal Approximation Capability argues that identity shortcuts free parameters to model only the correction. We borrow this as a design principle about parameterization, not as a theorem that transfers to code.
The closest chart work constrains the design.
- ChartM3 measures edits with ΔSSIM, a structural-similarity difference, across 10 chart types, but its models emit full scripts.
- ChartMaster widens chart diversity with ReChartPrompt-240K and reinforcement learning against a visual-similarity reward. It uses a ResNet-18 only as a frozen reward feature extractor, and it does not study test-time cost.
Our proposal differs from all of these:
- It changes what is distilled (a diff corrector, not a better first pass).
- It changes what is trained on (the student's own drafts).
- It changes where gains are measured (stratified by chart-type rarity).
Research Program
Teacher loop. We first build an AMACE/CITL-style teacher loop for Matplotlib chart-to-code. A generator drafts code, a sandbox renders it, a strong vision-language critic compares the render to the target chart image and writes a critique, and an improver rewrites the code. The loop runs for up to three rounds with early stopping.
Training data. Running this loop over ReChartPrompt-240K-style training charts yields triples: draft code, draft render, accepted final code. We convert each triple into a minimal edit. If $c_0$ is the draft and $c^*$ the loop's accepted code, the target is the line-level diff $\Delta$ such that $c^* = c_0 \oplus \Delta$, where $\oplus$ means "apply the patch." Drafts the loop accepts unchanged are kept with $\Delta = \varnothing$. This lets the corrector learn the identity explicitly, much like initializing a residual branch near zero.
Corrector. The corrector is a mid-sized vision-language model fine-tuned to emit $\Delta$ given $(c_0, \text{render}(c_0), \text{target image or instruction})$. We train two variants:
- one on drafts produced by the teacher's generator only;
- one on drafts produced by the deployed student generator, with the teacher loop run on those drafts to obtain repairs (the exploration-distillation variant).
Inference. The deployed system makes one generator call, one render, and one corrector call. An optional second corrector application is reported separately as a cost-quality point.
Analysis. We characterize what the corrector learns:
- the distribution of edit sizes;
- which code regions are edited;
- whether edits learned on common chart types fix the same error classes on rare ones.
Baselines
The naive reference is the generator's draft with no refinement, which defines zero gain.
The first strong competitor is generator distillation in the style of Sansford et al.. The same generator is LoRA-fine-tuned on the teacher loop's final code for the same training charts, then decoded in one pass. This is the most direct prior answer to "compress the loop," so it is the central comparison. It gets the same training charts, the same teacher outputs, and a matched number of fine-tuning steps and adapter parameters. Beating it means recovering a clearly larger fraction of the teacher's gain while staying within two model calls. Its reported 24.7% recovery comes from frontend HTML, not charts. For charts we will measure it ourselves.
The second competitor is a strong single-model chart generator trained with visual-reward reinforcement learning, as in ChartMaster. It reports a 93.8% execution success rate on ChartMimic, against GPT-4o's 93.2% reported in ChartMimic. On the same benchmark GPT-4o reaches 81.2 Direct Mimic score and InternVL2-Llama3-76B reaches 58.5. No reported ChartMaster Direct Mimic score was found on Lacuna, so we will reproduce or approximate its training at the same model scale. We will then test whether our corrector still adds gain on top of it, which is the fair framing since the two methods are complementary.
The third reference is the full teacher loop with early stopping, run like AMACE. It is the quality ceiling and the cost ceiling, and we report its calls, tokens and latency per chart.
For chart editing, the reference is Qwen2-VL fine-tuned on ChartM3-Train, which reports ΔSSIM 57.88 (from 6.5 zero-shot) in ChartM3. Here the corrector is applied after that editor.
Two ablations isolate the proposal's ingredients:
- A full-rewrite corrector has the same inputs and training data but emits the complete revised script instead of a diff. It isolates the residual parameterization.
- An image-blind diff corrector isolates the visual conditioning, mirroring the 1.5% "refine without critic" result.
Why not just…?
Why not just fine-tune the generator on the loop's outputs? This is the strongest alternative and the one the proposal must beat outright. The only direct evidence, from frontend code, is that it recovers 24.7% of loop gains (Sansford et al.). Our hypothesis is that it asks the first pass to anticipate errors it cannot see, because the render does not exist yet. If on ChartMimic the distilled generator matches the diff corrector's recovered fraction at equal cost, the proposal fails and we will report that.
Why not just sample K drafts and pick the best render? Best-of-K with a visual-similarity reranker is simple and often effective, but its cost grows linearly in K, which is exactly the overhead the ensemble literature tries to remove. We will compare it at matched generated tokens. A diff corrector emits short patches, so for a fixed token budget K is small. If best-of-K with K around 2 to 3 matches the corrector, the residual corrector adds nothing beyond a reranker, and we would redirect the work toward reranking.
Why not just train a better generator with more diverse data and visual reinforcement learning? ChartMaster shows that this route raises single-pass quality, and it is a genuine threat on common chart types where drafts may already be near-perfect. It is less clear that it helps rare types with few examples. The decisive test is stacking: apply the corrector to a ChartMaster-style generator and measure the remaining gain, stratified by chart type. If the gain vanishes everywhere, the corrector is redundant with better training.
Why not just let the corrector rewrite the whole script? Full rewrites can silently break working code and cost tokens proportional to script length. The full-rewrite ablation measures exactly this, through regression rate and output tokens. If rewrites are as safe and as accurate, the residual framing is cosmetic.
Proposed Experiments
Benchmarks. The main benchmark is ChartMimic Direct Mimic, chosen for its 22 chart categories and 201 subcategories. We split categories into a head (bar, line, pie, simple scatter) and a tail (heatmap, violin, box, plot-in-plot, multi-panel, and other rare types). Secondary benchmarks are:
- ChartMimic Customized Mimic;
- the ChartM3 editing test set across its 10 types, scored with ΔSSIM;
- ChartQA:augmented, evaluated through the AMACE protocol to connect to its reported 29.97%.
Metrics. - Benchmark score. - Execution success rate. - Recovered fraction, defined as (system minus draft) divided by (teacher loop minus draft), which is directly comparable to the 24.7% figure. - Regression rate: the share of charts whose score drops after correction. - Cost: model calls, generated tokens, and wall-clock time per chart.
Decisive checks. Five results settle the claim.
- Diff corrector vs. distilled generator, by chart-type tier. The proposal succeeds if the diff corrector recovers markedly more of the teacher's gain than the LoRA-distilled generator, especially on tail types, at two model calls.
- Diff vs. full rewrite. The diff corrector should show a lower regression rate and fewer output tokens at equal or better score.
- Exploration distillation. Training on the student's own drafts should beat training on teacher drafts. Following Liu et al., we expect the gap to widen when the student generator is weaker than the teacher's.
- Held-out chart types. This is the direct test of the residual-transfer hypothesis. We train the corrector with an entire tail category removed (for example, all violin and box plots). If corrections are shared local fixes, it should still help on the held-out type; a distilled generator trained with the same omission should not. A null result here would falsify the explanation even if the overall numbers look good.
- Stacking. The corrector is applied on top of a visual-reinforcement-learning generator and on top of the ChartM3 fine-tuned editor, to establish whether it is complementary to, or redundant with, better first-pass training.
References
- AMACE: Automatic Multi-Agent Chart Evolution for Iteratively Tailored Chart Generation
- Vision-Guided Iterative Refinement for Frontend Code Generation
- Distilling Knowledge for Search-based Structured Prediction
- Characterizing ResNet's Universal Approximation Capability
- ChartM3: Benchmarking Chart Editing with Multimodal Instructions
- ChartMaster: Advancing Chart-to-Code Generation with Real-World Charts and Chart Similarity Reinforcement Learning
- ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation