Painting the Error: Pixel-Space Feedback for Visual Code Repair

When a vision-language model writes code that draws something, such as a chart, a fractal, or a diagram, the natural way to fix its mistakes is to run the code, look at the picture, and try again. Current systems do the "look" step by asking a model to describe the differences in words: "the bars should be blue," "the left branch is too short." This proposal tests a different signal. The renderer compares its output with the target and paints the disagreement back onto the image as a color-coded error map: missing ink in one color, extra ink in another, misplaced elements outlined. The code-writing model then repairs its program while looking at that map. The idea borrows from work that turned every vision task into "produce an image." Here the task being turned into an image is error localization inside a program-repair loop.

The bet is that words are the wrong medium for spatial and numerical errors. Evidence for this already exists: verbal reasoning before coding lowers accuracy on visual-mathematical program synthesis, because narrative fails to pin down coordinates. Verbal-critic refinement loops nonetheless remain the norm. We also ask a question the closest prior systems skip: does the ranking of feedback types (painted, raw, verbal, none) stay the same across model families, or is it an artifact of one or two backbones?

Core Claim

For visual code repair where the target image is part of the input, spatially grounded feedback painted in pixel space produces larger and more reliable repair gains than natural-language difference descriptions at equal render and token budgets. On line-art tasks a deterministic overlay is enough. On charts, where small shifts make raw pixel differences useless, a small learned "discrepancy painter" trained in the Painter style is needed. The claim covers both the average gain and its consistency across at least five model families. It would be falsified if verbal critics match painted feedback, if best-of-N sampling at matched budget matches both, or if the ordering of feedback types flips between families.

Grounding in Prior Work

Three lines of work supply the motivation.

The first is diagnostic and shows the problem is reasoning, not seeing. MATH-Vision attributes 42.2% of GPT-4V failures to reasoning after correct recognition. FractalBench asks models to reconstruct fractals with a four-command turtle interface (move, turn, pen up, pen down) and scores the result by pixel overlap with the target. It finds that 76.1% of programs run but only 4.2% succeed, with branching structures below 2%, and that reasoning-then-code prompting hurts. FractalBench is strictly one-shot, so whether a feedback loop breaks the branching ceiling is open. We take its benchmark and its negative result on verbal reasoning as our starting constraint.

The second line supplies the loop we modify. ChartIR renders chart code and has a model describe differences in text before revising. Vision-Guided Iterative Refinement for Frontend Code Generation adds a visual critic and a code critic for HTML/CSS, and shows that refining without a critic gives only 1.5% while the critic loop gives up to 17.8%. Canvas-of-Thought lets a model edit a mutable DOM tree, renders it in a headless browser, and has a critique agent compare render and query, but it tests only GPT-5 and Gemini 2.5 Pro. All three keep the critique in language or in a separate agent's judgment. None hands the generator a pixel-aligned map of where it is wrong, and none checks whether its gains hold across open-weight families. ChartMimic observes informally that feeding a render back helps GPT-4o, but never formalizes the loop.

The third line supplies the representation. Painter and InstructCV show that heterogeneous outputs such as masks, depth, and boxes can all be painted as RGB images by one model. We reuse that encoding for a new output type: a categorical error map. Painter's fixed 448×448 resolution and InstructCV's roughly 5 seconds per sample on an A100 both constrain us. We therefore use a small masked-image-modeling painter, not a diffusion model, and run it only on charts.

MSRL is the strongest training-time competitor. It uses reinforcement learning with execution and rendered-judge rewards on one 7B model, and offers no test-time repair. Finally, Beyond Probabilities shows that conclusions drawn from a narrow model set often fail to transfer. This motivates treating cross-family consistency as a primary outcome rather than an appendix.

Research Program

The loop is fixed across all conditions. The model sees the target image and writes a program, which is executed. The render is compared with the target, a feedback artifact is produced, and the model revises. A best-so-far rollback keeps the highest-scoring program, which guards against the known pattern of edits that make things worse. The budget is a fixed number of renders, with five turns as the default, and a fixed context budget. Old programs are pruned so that only the current program, its render, and the latest feedback remain. Only the feedback artifact varies between conditions.

On FractalBench the painted feedback is deterministic. The render and target are binarized and overlaid on one canvas, with target-only pixels in red, render-only pixels in blue, and agreement in gray. The intersection-over-union value (the overlap measure FractalBench uses) is printed in a corner. This requires no learning, because line art on a blank canvas has no alignment problem.

On ChartMimic, raw pixel differences are dominated by sub-pixel shifts and anti-aliasing. Here we train a discrepancy painter. We take programs from Chart2Code-3M (the data used by MSRL) and apply typed mutations: change a series color, perturb a data value, drop a legend, swap a marker, shift an axis limit, or edit a label. Rendering the reference and the mutant, together with each element rendered in isolation, yields exact masks of the changed regions. A Painter-style model is trained to map the pair (reference, candidate) to an RGB canvas in which each error type has a fixed color, drawn over the candidate. Outside the painter, the loop is the same as on FractalBench. Separately, we report the painter's mask accuracy on held-out mutations so that critic quality is measured rather than assumed.

The cross-architecture study runs every condition on at least five families. Open-weight models span different vision connectors and language backbones: Qwen2-VL, InternVL2 at two scales including the 76B model ChartMimic reports, and a LLaVA-style model. Proprietary models are GPT-4o and Gemini 2.5 Flash, the best FractalBench model. The analysis asks whether the per-family ordering of feedback conditions agrees across families, measured by rank correlation, and whether painted feedback narrows the gap between open and proprietary models.

Baselines

The naive reference is one-shot generation, which matches each benchmark's original protocol. On FractalBench the reported numbers are an overall success rate of 4.2%, Gemini 2.5 Flash at 14.8%, GPT-4o at 3.5%, and below 2% on branching families (FractalBench). On ChartMimic Direct Mimic, GPT-4o reaches 81.2 and InternVL2-Llama3-76B reaches 58.5 (ChartMimic). We rerun all families ourselves with identical prompts rather than relying on these numbers.

The second baseline is critic-free self-refinement. It is the same loop in which the model sees only its own render and the target, with no difference artifact. This separates "seeing your output" from "being told where it is wrong." Vision-Guided Iterative Refinement reports only a 1.5% gain for refinement without a critic on WebDev Arena, which suggests this baseline will be weak, but we measure it.

The strongest test-time competitor is ChartIR-style verbal difference feedback. The same base model, or GPT-4o as a stronger critic in a separate arm, describes differences across axes, colors, style, grid, and text, and the generator revises. ChartIR reports a GPT-4o-judged score of 7.15 on ChartMimic with GPT-4o and 3.86 with Qwen2-VL. Those numbers are on a 10-point judge scale, not the ChartMimic 100-point score, so we reimplement ChartIR and report both metrics. Fairness means the same turn count, the same rollback, and verbal critiques allowed roughly the token budget of the painted image's visual tokens.

The strongest trained competitor is MSRL, which reports 83.8 high-level score and 96.5% execution rate on ChartMimic with a 7B model (MSRL). If weights are available, we apply our loop on top of it to test whether test-time painted feedback still adds anything after reinforcement learning. Otherwise we compare against its reported number with that caveat stated.

Ablations isolate the key ingredient. On charts, raw pixel-difference overlay is compared with the learned painter. A painted map that has lost its error-type colors (binary only) is compared with the typed map. The painted map is also converted to text by a template ("blue region at x 120–180, y 40–60: extra ink"), which tests whether pixel form matters or only localization does. Beating the strongest competitor means a statistically significant gain in FractalBench success and ChartMimic high-level score over verbal feedback at matched budget, in a majority of families, with no family showing a significant loss.

Why not just…?

Why not just use a verbal critic, as ChartIR does? This is the incumbent design. Our hypothesis rests on FractalBench's finding that verbal reasoning degrades coordinate-level accuracy. The direct comparison at matched budget decides it, and the template-to-text ablation tells us whether any win comes from localization or from the pixel medium itself. If text-rendered localization matches the painted map, we will report that the gain comes from localization and that pixel form is not needed.

Why not just sample N programs and keep the best by pixel overlap? This is the most serious threat on FractalBench, because the scoring signal is available at test time and branching fractals may be solved by luck more often than by repair. We run best-of-N with N equal to the total render budget of the loop. If best-of-N matches painted repair, the models are not actually using feedback, and we will say so. That would itself be a useful finding about the branching ceiling. We also report the combination of a few samples, each repaired, to see whether repair and sampling add up.

Why not just train with reinforcement learning, as MSRL does? Training-time optimization and test-time repair are not exclusive. MSRL is validated on one backbone and requires rendering and a judge model inside training. Painted feedback needs no training of the generator and can be applied to any family. The experiment stacking our loop on an MSRL-style model shows whether test-time localization adds anything once the generator is already strong. If the gain vanishes there, the method's value is limited to untrained or weaker models, and we will scope the claim accordingly.

Why not just use the raw pixel difference and skip the learned painter? On FractalBench we do exactly that. On charts we expect raw differences to light up nearly every edge after small shifts. The raw-versus-painter ablation and the painter's held-out mask accuracy settle whether the learned component is needed.

Decisive Checks

Three results would decide the proposal.

The first is whether painted feedback lifts branching-fractal success on FractalBench, currently below 2% one-shot, measurably above both verbal feedback and best-of-N at an equal number of renders. This is the cleanest test of whether localized error signals help models build recursive structure.

The second is whether, on ChartMimic, the learned painter beats both raw differences and ChartIR-style critiques on the low-level text, layout, type, and color scores. The color and layout scores are where spatial feedback should help most.

The third is whether the ordering of feedback conditions holds across all five families with positive rank agreement, and whether painted feedback shrinks the gap between open and proprietary models. If the ordering holds only for proprietary models, the method inherits the narrow-scope problem it set out to test, and that negative result would be the main finding.