When a vision-language model turns a screenshot or a software diagram into code, it first shrinks the image to a small fixed grid. Tiny details do not survive: a - that marks a private method, a dashed arrow that means "implements" rather than "extends", or a 10-pixel icon button. The model does not report that it cannot see them. It writes the most typical code instead, such as public methods, a generic submit button, or common CSS values. This is the code-generation version of a familiar failure: a captioning model that "sees" a refrigerator in every kitchen because refrigerators usually appear there.
We propose to catch these guesses in the model's own output and repair them locally. For each small piece of the generated code, we measure how much the model's confidence depends on the image. Pieces the model would have written equally well without the image are flagged as guesses. We then crop the corresponding source region at full resolution, re-read it, and rewrite only that piece. The bet is that a large share of errors in image-to-code generation are not reasoning failures. They are guesses filling in detail that downscaling destroyed, and spending extra visual tokens only where the code guessed should beat both uniform high-resolution tiling and render-and-compare refinement at the same compute.
Core Claim
Three bodies of work each see part of this problem, but none connects the parts.
Hallucination research shows that language priors override weak visual evidence. POPE found LLaVA-style models answering "yes" to object-existence questions up to 99% of the time. HalTrapper shows that attention drifts from image tokens toward previously generated text as responses grow. This work motivates our mechanism, but it studies natural objects in captions and fixes errors at decoding time. Decoding-time fixes cannot restore pixels that downscaling has erased.
Resolution research shows that the encoder's input size is the binding limit. UReader's shape-adaptive cropping raised DocVQA from 22.0 to 65.4. DCGen shows that a failed webpage segment, re-fed at native resolution, is often coded correctly with no change to the model. Both crop by fixed geometry, not by where the model was wrong.
Image-to-code work names the symptoms without diagnosing them as prior fill-in:
- From Diagrams to Code reports that large diagrams lose visibility modifiers and guard labels.
- Design2Code reports a "clean-up effect": models modernize pages by replacing what is shown with what is typical.
Our claim has two parts. The first is diagnostic: under a controlled resolution sweep, code errors in image-to-code tasks shift toward the prior-typical value, and the shift is concentrated in content-bearing leaves of the program's syntax tree. Examples of such leaves are text literals, visibility modifiers, relation types, colors and sizes. The second is methodological: a per-leaf image-dependence score, combined with region anchors the model emits alongside its code, is a better trigger for where to re-inspect than global render comparison, uniform tiling, or test failure.
To define the score, let $s$ be a leaf span of generated code, $I$ the input image as the model sees it, $x$ the text instruction, and $\varnothing$ a blank image. The image-dependence of $s$ is
$$d(s) = \log p(s \mid I, x) - \log p(s \mid \varnothing, x).$$
A content-bearing leaf with low $d(s)$ is one the model would have written with no image at all. We treat such leaves as prior fill-in and candidates for a second look.
Research Program
The work has three parts: a probe, a method, and a cost analysis.
The probe: a code-level analogue of POPE. We render UML class diagrams, flowcharts and simple webpages from programs whose content we control. In each, we deliberately plant atypical values: private fields, unusual relation arrows, non-default colors, and rare but legible labels. Each planted value has a typical counterpart, so an error toward the typical value can be scored as prior fill-in rather than random noise. Following POPE's random, popular and adversarial splits, the planted values range from mildly to strongly atypical.
We sweep the input resolution and the target's pixel footprint. For each condition we measure two things: the prior-default rate (errors that land on the typical value) and how well $d(s)$ separates correct leaves from incorrect ones, measured as AUROC. This tests the diagnostic claim on its own terms, independent of whether the method works.
The method: anchored generation with prior-triggered zoom. We fine-tune an open 7B model lightly so that, while writing code, it attaches a bounding-box comment to each leaf naming the source region it was read from. The base is Qwen2-VL-7B, the backbone of M²-Coder, which already supports box outputs. Supervision comes from our rendered data, where these correspondences are known by construction.
At inference, two triggers select regions to re-inspect:
- Distortion trigger. Leaves whose $d(s)$ falls below a threshold are flagged as likely guesses.
- Omission trigger. Omitted elements produce no tokens to score. This trigger therefore finds source regions containing ink or edges that no anchor covers.
Each flagged region is cropped from the original image at native resolution. The model then regenerates only the affected syntax-tree node, conditioned on the crop, the surrounding code, and the global low-resolution view. The rest of the program is held fixed. This local edit is our answer to the repair-coupling failure that RubSE documents, where a CSS fix in one place breaks layout elsewhere. Where tests exist (M²EVAL, HumanEval-V), failing tests can add flags: the traceback is mapped to the syntax-tree nodes involved and, through their anchors, to image regions.
The cost analysis. We count visual tokens and wall-clock time per task. The claim only holds if accuracy improves at a matched budget, not only an unlimited one.
Baselines
Direct prompting. The naive reference is the same base model given the full downscaled image in a single pass, with the same decoding settings. Any proposed gain must first appear against this.
Uniform high-resolution tiling. This is the strongest competitor on the resolution question. It is the UReader and Monkey approach: split the image into native-resolution tiles plus a global view. We run it on the same base model at two budgets, matched visual tokens and unlimited tiles. Beating it at matched tokens is the result that matters. Matching it at a fraction of the tokens would also count.
Geometry-based decomposition. DCGen recursively slices pages along whitespace and borders and reassembles the code. It reports +15% CLIP similarity and +8% code similarity over direct prompting on its Tranco-derived set. We reimplement it on the same base model and crop budget, and apply its slicing to diagrams as well, where we expect it to break.
Render-and-compare refinement. This is the strongest refinement-style competitor for webpages. UI2Code^N is a 9B model trained with reinforcement learning on pairwise visual comparisons. It reports 88.6% on Design2Code-HARD, versus 41.9% for Qwen2.5-VL-72B. We cannot match its training compute. We therefore compare in two ways:
- Against a training-free render-diff loop on our base model with an equal number of refinement rounds, plus RubSE-style single-rubric repair.
- Plugging our trigger into UI2Code^N's model, if its weights are available, to test whether the two are complementary.
Diagrams and tests. On M²EVAL, the reference points are M²-Coder at 25.3% Pass@1, Qwen2-VL-72B at 24.7%, and GPT-4o at 49.7%, all as reported in From Diagrams to Code. On HumanEval-V, the strongest reported loop is Claude 3.5 Sonnet with test-execution refinement, which rises from 36.8% to 55.3% over four rounds (HumanEval-V). We run the same four-round test loop on our base model without zoom.
Ablations. Each isolates one ingredient:
- Random zoom: random regions at the same crop count. Isolates the value of the trigger.
- Render-diff-only zoom: triggers from render comparison alone. Isolates the value of $d(s)$.
- Contrastive decoding without zoom: penalizes low-$d$ tokens at decoding time, in the spirit of HalTrapper's contrastive decoding. Separates "detecting the guess" from "recovering the pixels".
- Full regeneration after zoom: rewrites the whole program instead of one node. Isolates the local edit.
Why not just…?
Why not just tile the whole image at high resolution? For moderate image sizes this is a real threat. If our method only matches tiling at equal tokens, the trigger has added nothing and we will say so. The file's evidence suggests tiling degrades as images grow. Token counts explode and crowd the context window. Diagram-to-code models are reported to lose fine marks on large diagrams even with current visual front-ends (From Diagrams to Code). Longer contexts also pull attention away from image tokens (HalTrapper), so more tiles may add as much drift as evidence. The matched-budget comparison across image sizes settles this.
Why not just suppress hallucinations at decoding time? Contrastive or summary-guided decoding can stop a model from emitting a typical-but-wrong token. It cannot tell the model what the correct token is if the pixels were erased. At best it produces abstention or a different guess. The contrastive-decoding-without-zoom ablation measures this directly. We predict it lowers the prior-default rate on the probe but recovers far less of the correct value than zooming does.
Why not just run tests or render-and-compare loops? Test feedback is powerful when it exists: HumanEval-V's four-round loop added 18.5 points for Claude 3.5 Sonnet. It does not say where in the image to look, though, and most webpage and class-design tasks lack tests. Global render comparison drives naive self-refinement into collapse after two to three rounds (RubSE) and into cosmetic oscillation (UI2Code^N). We treat tests as an additional trigger rather than a rival. The test-loop baseline shows whether $d(s)$ adds anything once tests are available.
Why not just use OCR or a layout detector? Text-only pipelines score 0% on both HumanEval-V and M²EVAL, because the logic lives in geometry and relations. Text-only DeepSeek-V3 scores 0.0% on M²EVAL. OCR also drops symbols like +/- and arrowhead styles, which are exactly the targets here. We include an OCR-augmented prompt as a sanity baseline on the probe only.
Proposed Experiments
We evaluate on four test sets:
- Design2Code and Design2Code-HARD, scored by Block-Match, text, position and color similarity, and CLIP similarity.
- M²EVAL, scored by Pass@1 across its 10 languages.
- HumanEval-V, scored by Pass@1.
- Our rendered probe, scored by prior-default rate and the AUROC of $d(s)$.
All methods share the base model, decoding settings, number of refinement rounds, and visual-token budget unless a baseline is explicitly run at unlimited budget.
Four checks decide whether the idea holds:
- Is fill-in real? On the probe, the prior-default rate must rise as target footprint shrinks. $d(s)$ must separate wrong from right leaves better than token entropy alone. If errors are random rather than prior-typical, the diagnostic claim fails and the method reduces to ordinary active zoom.
- Is the trigger useful? At matched visual tokens, prior-triggered zoom must beat random zoom, render-diff-only zoom, and uniform tiling on Design2Code-HARD Block-Match and on M²EVAL Pass@1. Gains should concentrate on small elements. The baseline here is GPT-4o's drop from 93.0 to 56.6 Block-Match between the standard and HARD splits. We stratify by element size to test this.
- Does the local edit prevent regression? Node-local regeneration must avoid the round-three degradation that RubSE documents for naive loops. We track it with per-round score trajectories.
- Does the gain stop where the theory says? HumanEval-V serves as an honest boundary. Its reported failures are spatial transformations such as rotation and path tracing, not invisible details. We therefore predict small gains there. A large gain would mean our story about why the method works is wrong, even if the method works.
References
- Evaluating Object Hallucination in Large Vision-Language Models
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model
- Divide-and-Conquer: Generating UI Code from Screenshots
- From Diagrams to Code: Multilingual Programming with Visual Design
- Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering
- Rubrics as Visual-Repair Context for Self-Evolving UI-to-Code Generation
- Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models
- UI2Code$^{\text{N}}$: UI-to-Code Generation as Interactive Visual Optimization
- HumanEval-V: Evaluating Visual Understanding and Reasoning Abilities of Large Multimodal Models Through Coding Tasks