Execution as a Witness: Telling Apart Instructions and Noise in Image-to-Code Generation

When a multimodal model turns a chart or a webpage screenshot into code, the code can be run and its output rendered as an image. That rendered image is a precise record of what the model actually used from its input. This proposal uses that record to answer two questions. Which parts of the accompanying text changed the model's output? And did those changes follow the image, follow a legitimate request, or follow a distraction? Prior work has shown both effects separately. Plot2Code found that adding a text description to a plot image often makes the generated code less likely to run, which it called the "Instruction Paradox" (Plot2Code). EEE-Bench showed that a misleading caption pulls GPT-4o away from what a circuit diagram shows (EEE-Bench). Neither explains the mechanism. Neither offers a fix that keeps the useful part of the text.

We propose an inference-time method. The model generates code under several reduced versions of the prompt, every candidate is executed, and the one kept is the one whose rendered output both matches the image and verifiably satisfies the explicit requests in the text. Alongside the method we build a controlled noisy-context version of existing chart-to-code and screenshot-to-code benchmarks. With it we test a hypothesis about the gap between open-weight and proprietary models: under noisy context, a substantial share of that gap comes from open models following irrelevant or conflicting text, and execution-checked context selection can recover that share. It cannot recover the share that comes from weak visual perception.

Core Claim

Rendering acts as a free oracle for attributing context. Take a run with a text span and a run without it, and compare the two renders. The difference shows exactly what that span did to the output: a changed color, an added subplot, a broken axis, or a crash. The source image and a structured list of the explicit edits requested in the text then give an external standard for judging each change. We claim three things.

First, this attribution explains the Instruction Paradox. We predict that most execution failures in Plot2Code's image-plus-text setting trace to spans that conflict with the image or add narrative detail, not to spans that request legitimate edits.

Second, a selector that picks among context-ablated candidates by an execution-checked score beats spending the same compute on plain resampling. It also beats prompt summarization and dropping the text entirely.

Third, the open-versus-closed gap is larger under contextual noise than on clean inputs, and the selector narrows the noise portion of the gap without closing the clean portion. That last result is what would distinguish "open models are distractible" from "open models cannot see."

Research Program

The benchmark. The benchmark adds controlled noise to inputs whose clean reference code already exists. This means execution-based scoring stays exact.

We start from three sources: ChartMimic's chart–instruction–code triplets, which include a "Customized Mimic" setting where the text legitimately asks for changes (ChartMimic); Plot2Code's image-only and image-plus-description settings; and Design2Code's webpage screenshots, including the long, complex HARD subset (Design2Code).

On these inputs we inject three families of noise, each borrowed from a paper that documented it in a different setting:

  • Narrative padding. Backstory and irrelevant nouns wrapped around the actual request, in the style that Less is More showed misleads text-only code models.
  • Conflicting claims. Text that misstates something visible, such as the number of bars or an axis range, in the style of EEE-Bench's false captions.
  • Real-capture degradation of the input image. Blur, glare, perspective tilt and background handwriting, following the categories in MathReal.

Legitimate edit requests are kept and labeled separately. A good system should be sensitive to them and insensitive to the noise. This pairing addresses the "information-noise dilemma" that the noise-filtering literature leaves open: filtering too aggressively throws away real requirements.

The method. For each input, a language model first extracts from the text a short structured list $E$ of explicit, checkable edit requests, such as "use a log y-axis" or "make the header blue." It then builds a small set of context variants: the full text, an LLM summary, the edit list $E$ alone, and no text. The code model generates candidates under each variant, and every candidate is executed in a sandbox. Candidates that crash are discarded. Each survivor $c$ with render $R(c)$ is scored as

$$ S(c) = \mathrm{sim}_{\neg E}\big(R(c), x\big) + \lambda \cdot \mathrm{comply}_E(c), $$

where $x$ is the input image. $\mathrm{sim}_{\neg E}$ is an element-level similarity between render and input image, computed only over attributes the edit list does not touch: text, layout, chart type and color, in the spirit of ChartMimic's low-level metrics and Design2Code's Block-Match. $\mathrm{comply}_E(c)$ is the fraction of requested edits that a rule-based check finds in the code or render. $\lambda$ is a single weight tuned on a validation split. The highest-scoring candidate is returned.

Attribution falls out of the same runs. If the full-text variant and the edit-list-only variant render differently on an attribute outside $E$, the extra text caused that change. This gives a per-example, execution-grounded label of how distracted the model was.

Analysis of the gap. We report open-versus-closed gaps separately on clean, text-noised and image-noised inputs, before and after selection. The disparity hypothesis predicts three outcomes:

  • On clean inputs, selection barely moves the gap.
  • Under text noise, it moves the gap a lot.
  • Under image noise, it moves the gap little. When the model misreads the image, scoring candidates against that image cannot fix the misreading.

This last prediction is a deliberate falsification target. It fits MathReal's finding that supplying the ground-truth text largely restores accuracy, which points to perception as the bottleneck under visual noise.

Optional training step. If the selector works, its chosen outputs can be distilled into the open model with low-rank fine-tuning. This follows the distillation recipe of Vision-Guided Iterative Refinement, which reports that distillation captured 24.7% of its best refinement gains. Distillation is secondary. The core result must hold at inference time.

Relation to Prior Work

The closest existing systems all use execution or rendering feedback on clean specifications.

MSRL trains a 7B chart-to-code model with reinforcement learning on execution and rendered-image rewards. UI2Code^N does pairwise visual-comparison reinforcement learning for webpages. The critic-in-the-loop system of Vision-Guided Iterative Refinement renders HTML and asks a vision model to critique it. VDebugger feeds line-by-line execution traces of visual programs to a critic. These papers supply our method's main ingredient, render-and-compare. None of them varies the context or asks which input span caused an output. Our proposal does not train a stronger generator. It uses execution to decide which parts of the prompt to trust.

SRank and VRank choose among samples by clustering their runtime behavior. We reuse that idea but cluster across deliberately different contexts rather than repeated samples of one prompt. That distinction is exactly what our ablations test.

The noise papers motivate the work. Less is More showed that summarizing narrative-heavy problems raises strict execution accuracy by 8.13% on APPS and 11.88% on CodeContests. It also showed only about a 2% gain on the hardest problems. That limit constrains our expectations: context cleaning cannot fix a missing skill. EEE-Bench and MathReal documented textual and visual distraction, but only in question answering, with no code to execute. Quantifying Cross-Modality Memorization shows that background visual noise gets absorbed during training and that multi-hop cross-modal links collapse to 11%. That is a warning that some image-noise failures sit in the model itself, out of reach of inference-time selection.

Baselines

Naive reference points. The do-nothing reference is the open model given the full noisy prompt with greedy decoding. The second naive reference is image-only generation, Plot2Code's "Direct Asking," which is the bluntest possible noise filter. On clean ChartMimic the reported open reference is InternVL2-Llama3-76B at 58.5 (Direct Mimic) and 64.7 (Customized Mimic) overall score, against GPT-4o's 81.2 and 83.2 with a 93.2% execution rate (ChartMimic). On Plot2Code, GPT-4V scores 7.83/10 and Mini-Gemini-8x7B-HD 6.08 in the image-plus-text setting (Plot2Code). No reported result was found on Lacuna for any model on noise-perturbed versions of these benchmarks, so all noisy-condition numbers will be produced here.

Strongest existing methods. The strongest generator is MSRL's 7B model, reported at a 96.5% execution rate and an 83.8% high-level score on ChartMimic (MSRL). On webpages the strongest is UI2Code^N, reported at 88.6% on Design2Code-HARD against 41.9% for Qwen2.5-VL-72B (UI2Code^N). Each is the strongest competitor because it already uses rendering signals and might already be robust. We will run them on our noisy splits with the full prompt, and as backbones under our selector, if their weights are available. If they are not, we use the strongest open checkpoint we can obtain and state this explicitly.

The strongest inference-time competitor is the critic-in-the-loop refiner (Vision-Guided Iterative Refinement). It reports a 17.8% gain over three cycles, and only 1.5% for refining without a critic. We give it the same number of generations and executions as our selector.

Fairness and ablations. All methods share the backbone, the sandbox and a fixed budget of $k$ generations and executions per item. The ablation isolating our key ingredient keeps the scoring rule $S$ fixed but draws all $k$ candidates from the full prompt by sampling, rather than from different context variants. A second ablation drops the compliance term ($\lambda=0$) to show what happens to Customized Mimic edits when only image similarity is used.

What counts as beating them. Beating the baselines means a higher overall score and execution rate on the text-noised splits at equal budget. It also requires no loss of edit compliance relative to the full-text greedy baseline, and no significant loss on clean inputs.

Why not just…?

Why not just drop the text and generate from the image? It removes all text noise and is free. But on Customized Mimic and Plot2Code's image-plus-text setting the text carries required changes, so an image-only system fails by construction on those requests. Our compliance metric will show this directly. If image-only matches our selector on the pure-mimic splits, we will report that honestly: the method's value then lies entirely in customized settings.

Why not just summarize the prompt, as in Less is More? Summarization was the strongest text-only fix, but it never looks at the image. A summary of a prompt that falsely says "five bars" still says five bars. Our conflicting-claims split targets exactly this case. Summarization is also one of our context variants, so if it wins we will see it selected almost every time. That would be an informative, if deflating, outcome.

Why not just sample more from the full prompt and pick by consensus or render similarity? This is the real threat. Repeated samples might already avoid distractors often enough that a good scorer finds a clean one. The same-scorer, same-budget ablation settles it. Our prediction is that full-prompt samples share the same distraction, since the noise is in every sample's input, so their renders agree on the wrong attribute and consensus picks the error. If plain resampling closes most of the gap, the context-variation idea is not doing the work, and we would reframe the contribution as the attribution analysis alone.

Why not just train with noise-augmented data under MSRL-style rewards? With clean targets, augmentation directly teaches invariance, and it may well beat inference-time selection on in-distribution noise. We treat it as a complementary upper reference rather than a competitor we expect to beat. It needs paired clean targets and training compute; our method needs neither. The test that matters is held-out noise types: we train augmentation on two noise families and evaluate on the third.

Decisive Checks

Three results would make the case.

The paradox is explained. On Plot2Code, at least half of the execution failures that appear when moving from image-only to image-plus-text are attributed, by render differencing, to non-edit spans. If instead most failures trace to legitimate edit spans, the paradox is a capability problem, not a noise problem, and our framing is wrong.

Selection beats resampling. At equal budget, the context-variant selector beats same-scorer full-prompt resampling on the narrative and conflicting-claims splits of ChartMimic and Design2Code. It must also keep edit compliance at or above the full-text baseline. If it does not, the context-variation idea adds nothing.

The gap moves as predicted. Selection narrows the open–proprietary gap substantially under text noise, marginally under image degradation, and negligibly on clean inputs. If it narrows the clean gap as much as the noisy gap, it is just a generic reranker. If it narrows nothing, the open models' failures are not context-driven at all.