Render-to-Source Feedback for Visual Program Synthesis

When a vision-language model is asked to write a program that redraws an image, such as a fractal, a chart, or a webpage, the program usually runs. It usually draws the wrong thing. On FractalBench, 76.1% of generated turtle-graphics scripts execute cleanly, yet only 4.2% reproduce the target fractal (FractalBench). The interpreter has nothing to say about such programs because nothing crashed. Current repair loops therefore hand the rendered output to another vision-language model and ask it to describe what looks wrong. That critic is itself a known source of hallucination: models acting as visual judges invent defects that are not there and misread layouts (Chen et al., Kaiyi Huang). The silent bug ends up being diagnosed by an unreliable witness.

We propose to take the critic out of perception entirely. The program is run under instrumentation that records which source line, and which recursion depth, produced every drawn stroke. The rendered image is then compared to the target with plain image processing, and every region of disagreement is traced back to the code that caused it. The result is feedback such as "strokes emitted by line 9 at recursion depth 3 run at 30°, while the target's ink in that region runs at 45°." Every number in that sentence is measured, not perceived. We then train a compact open model to act on this feedback inside a refinement loop. The bet is that precise, located feedback lets a 7B model cross the gap between running code and correct code, which frontier models do not cross in a single pass.

Core Claim

In image-to-code tasks the target image is the input, so a pixel-level comparison between the render and the target is a legitimate, non-hallucinating test-time oracle. What is missing is a way to turn that oracle's verdict into an edit. Our claim is that attributing disagreement regions to source locations through the execution trace is the ingredient that makes refinement work. A scalar similarity score is too coarse to say what to change. A free-text vision-language critique is too unreliable to trust.

We further claim that a compact model trained with reinforcement learning on multi-turn episodes using this feedback will beat both frontier models and compute-matched best-of-N sampling on FractalBench. The gains should concentrate on branching recursion, where every model tested so far scores below 2%. The gains should also hold up under a test that pixel matching alone cannot game.

Relation to Prior Work

Three papers already close a visual loop around code generation, and all three route diagnosis through a model's perception.

ChartIR has a multimodal model describe differences between the rendered and reference chart, then rewrites the script (ChartIR). Its authors note that it never consults programmatic signals.

Critic-in-the-Loop uses a vision-language critic plus a code critic for HTML (Vision-Guided Refinement). It shows that refinement without any critic gains only 1.5%, but it concedes that the judge is biased.

UI2Code^N trains a 9B specialist and replaces absolute judge scores with pairwise tournaments precisely because absolute scores are noisy (UI2Code^N). It still reports polishing oscillations on near-correct pages.

These papers motivate our work: they establish that rendered feedback helps and that perceptual critics are the weak link. We differ by removing perception from the diagnosis step.

The method borrows from two text-only lines of work.

SAC-Opt exposes silent errors in optimization code by back-translating code into statements and regenerating only the misaligned constraint (SAC-Opt). From it we take the idea of localized repair that leaves correct parts untouched.

IterGen backtracks to grammar symbols and crops the key-value cache when a semantic check fails (IterGen). From it we take symbol-level regeneration. Its checks are static, whereas our check is the rendered output.

ChartMimic's Code Tracer already hooks Matplotlib runtime objects to log text, layout, type, and color (ChartMimic). We reuse that kind of instrumentation, but as feedback for repair rather than as an evaluation score.

The specialization recipe follows GUI-Spotlight (GUI-Spotlight) and AutoTriton (AutoTriton). GUI-Spotlight trains a 7B model with supervised trajectories, then group-relative reinforcement learning with an auxiliary cross-entropy loss against collapse. AutoTriton uses the same family of policy optimization with execution rewards.

FractalBench also constrains the design. Its finding that verbal "reasoning-then-code" hurts, because the words drift from the coordinates, tells us the feedback must be numeric and anchored to code, not narrative.

Research Program

The tracer. We first build the tracer for FractalBench's four-primitive turtle interface. A thin wrapper around move, turn, pen_up, and pen_down records, for each stroke, its endpoints, heading, the calling line, and the depth of the Python call stack inside user-defined recursive functions.

The discrepancy module. This module works directly on pixels. It finds spurious ink (drawn but absent from the target) and missing ink (present in the target but not drawn). For missing regions it also estimates the target's local stroke orientation and scale with standard image gradients.

Attribution. Spurious ink is assigned exactly to the strokes that produced it. Missing ink is assigned to the nearest drawn strokes and their provenance, with the distance reported so the model can tell a confident attribution from a guess.

The feedback message. These measurements are compiled into a short structured message giving the line, the depth, the measured property, the target property, and the area affected. The message contains no adjectives, and there is no image for the model to misread.

Symbol-level editing. The policy is asked to rewrite only the function or statement span named in the message. Untouched spans are kept verbatim, which also cuts tokens relative to whole-file regeneration.

Training data. We generate fractals procedurally from randomized L-systems and recursive turtle programs across FractalBench's twelve families. Parameters are sampled so that no image overlaps FractalBench's 610 test images.

Training. We fine-tune Qwen2.5-VL-7B, the open model family ChartIR also adapts. Training starts with supervised data on single-turn and repair-turn examples. It continues with group-relative policy optimization over episodes of up to K repair turns, with the reward taken from the final render, and with the cross-entropy stabilizer from GUI-Spotlight.

Guarding against reward hacking. A pixel reward invites a degenerate solution: hard-coding segment coordinates to match the picture. FractalBench reports that models already drift toward flat loops and memorized pixel plots. We guard against this in two ways. First, the reward includes a held-out-depth term: the program, executed with its recursion depth parameter set one level deeper, must match the ground-truth fractal at that depth, which the policy never sees. Second, we cap the number of primitive call sites in the source.

Transfer to charts. We port the tracer to Matplotlib by attributing each artist's bounding box and properties to the line that created it. We then test on ChartMimic. Charts are the harder case, because the target's own properties (colors, positions) must be read from pixels, but color sampling and box overlap remain measurements rather than judgments.

Baselines

Naive single-pass reference. Frontier and open models generate once, as in the FractalBench protocol. The strongest reported number is Gemini 2.5 Flash at 14.8% accuracy under the 95% IoU threshold, with GPT-4o at 3.5% and branching structures below 2% for all models (FractalBench). We rerun these where APIs allow and quote the paper otherwise. Beating 14.8% with a 7B model is the minimum bar for the specialization claim.

Strongest existing refinement method. This is ChartIR-style iterative visual-difference refinement, where a strong multimodal model describes render-versus-target differences and the generator revises. We give it the same generator, the same number of renders per problem, and the same turn limit as our method. On ChartMimic Direct Mimic, ChartIR reports a GPT-4o score of 7.15 with refinement and Qwen2-VL 3.86 (ChartIR). Those scores come from an LLM-judge scale that differs from ChartMimic's own 0–100 overall score, on which GPT-4o reaches 81.2 single-pass (ChartMimic). We will report both scales and Code Tracer sub-metrics side by side rather than mix them. No reported result was found on Lacuna for any iterative refinement method on FractalBench, so there we run ChartIR's protocol ourselves.

Hallucination-mitigated critic. This is the same vision-language critic with RITUAL's random-transform dual input (RITUAL). It tests whether a better-behaved perceptual critic closes the gap.

Compute-matched best-of-N. We sample N independent programs and keep the one with the highest IoU against the input, at the same render and token budget as our K-turn loop.

Ablations isolating the key ingredient. At equal budget, the policy receives one of three signals instead of the attributed message: the scalar IoU only, the raw difference-overlay image, or the attributed message with line and depth fields removed. Separately, we compare whole-file regeneration against span-level regeneration, and reinforcement learning with the attributed loop against reinforcement learning with final IoU but no in-episode feedback. Beating a baseline means a higher held-out-depth-verified success rate, with a paired bootstrap interval over the 610 images that excludes zero.

Why not just…?

Why not just do best-of-N with the IoU oracle? This is the most serious threat. The oracle is free at test time, so selection alone may harvest most of the gain. FractalBench's numbers suggest that independent sampling rarely lands on correct branching programs, since success there is under 2% per sample. That is a hint, not proof. Our matched-budget comparison settles it, and we commit to the outcome. If best-of-N matches the attributed loop on branching families, the contribution reduces to the training recipe and the held-out-depth check, and we will report it that way.

Why not just fix the critic's hallucinations with existing decoding interventions? Methods like RITUAL and SHARP's steering vectors (SHARP) lower object hallucination on POPE and CHAIR. Those benchmarks ask about object presence in natural photos, not about angles and depths in recursive line drawings. The critic-plus-RITUAL baseline measures directly whether such a fix suffices. Our expectation is that it narrows the gap but cannot supply line-level localization, which no perceptual critic outputs.

Why not just reason in words before coding? FractalBench reports that reasoning-then-code lowers accuracy because the verbal plans hallucinate geometry. UI2Code^N finds a 3.1% gain from planning, but for macro layout. We include a reasoning-then-code arm for the same 7B model to confirm that the FractalBench finding holds after specialization.

Why not just reinforce on final IoU and skip the in-loop feedback? UI2Code^N and AutoTriton show that execution rewards alone can specialize compact models. The arm that uses IoU reward with no attributed messages tests whether the policy learns to repair without being told where. If it matches our method, attribution is unnecessary at training time, though it may still matter for sample efficiency, which we track as success versus training episodes.

Decisive Checks

The primary result is FractalBench accuracy at 95% IoU, overall and on the branching families, for the specialized 7B model with attributed refinement. It is compared against Gemini 2.5 Flash's reported 14.8% and against every matched-budget baseline above. Every success must also pass the held-out-depth test, and we report how many IoU successes fail it as a direct measure of pixel memorization masquerading as semantics.

The second check tests the plateau that Yan et al. observed for compiler-feedback loops (Yan et al.). We plot success against turn number for the attributed loop and for the perceptual critic. If attribution is the cause, its curve should keep rising past the turn where the critic's curve flattens or oscillates. We also count regressions, meaning turns where a previously correct region breaks, to test whether span-level editing avoids the error compounding reported by Srivastava et al. (Srivastava et al.).

The third check is transfer. We apply the Matplotlib tracer to ChartMimic Direct Mimic with the same compact model and report both ChartMimic's overall score and execution rate. ChartMimic shows open-weight models falling below 50% execution, so the bar is whether a 7B model closes the distance to GPT-4o's 81.2 more than ChartIR's refinement does.

The final check is a small audit of critic claims. We sample 200 feedback messages from the vision-language critic and 200 from the tracer, and verify each claim against the instrumented render. This quantifies how often the critic invents a defect.