Models that turn a webpage screenshot into HTML and CSS have to write out many exact values: every heading string, every hex color, every padding: 12px. Today they produce all of these from memory, one token at a time. The only evidence they have is a downsampled view of the image, and in that view small text, thin borders and one-pixel offsets have already been blurred away. This proposal splits the job in two. The code decoder keeps writing structure (tags, flexbox rules, nesting) as it does now. For literal values, it instead points at a region of the screenshot, and a deterministic operator measures the value (color, size, position, or a zoomed transcription of the text) from the original, full-resolution pixels. The value then goes into the code verbatim.
This joins two ideas that have not met. The first is the pointer-generator, a decoder that at each step chooses between generating a word and copying one from its input. The second is the visual resolution bottleneck. The pointer's "input" becomes screen regions rather than text tokens, and what is copied is a measurement rather than a token. The model therefore never has to recover fine detail through the low-resolution vision encoder at all. The bet is that much of the error on dense pages comes from this reconstruction step, and that a pointer can remove it in a single generation pass, without the multi-turn render-and-fix loops current systems use.
Core Claim
In screenshot-to-code generation, fine-grained failures on dense pages can be reduced by routing literal values through a cross-modal pointer over full-resolution screen regions plus exact measurement. Doing so should beat raising the encoder's input resolution with tiling, and it should beat post-hoc visual refinement, at matched compute. The gains should concentrate where the bottleneck predicts: small elements, color accuracy, and position accuracy.
The motivation comes from three findings. First, Design2Code, a 484-page real-webpage benchmark with element-level metrics, shows GPT-4o's Block-Match falling from 93.0 to 56.6 on its HARD subset and traces the drop to lost small elements. Second, UReader, which crops documents into native-resolution tiles, shows input resolution rather than language-model capacity limiting text-heavy understanding. Third, VSA, which parses screenshots into layout trees before emitting components, reports that its spatial parser misclassifies tiny icons and badges.
The method draws on pointer-generator work. Get To The Point introduced the soft generate-or-copy gate for summarization. Code Completion with Neural Attention and Pointer Networks used pointing to copy out-of-vocabulary identifiers. LargePiG showed that a pretrained model's own attention can serve as a training-free copy distribution. The closest prior work is LargePiG, and the contrast with it is sharp. Even on LLaVA, LargePiG copies only from textual prompt tokens, and it copies tokens. This proposal copies from visual regions, and what it copies is a value computed from pixels the encoder never saw at full fidelity.
Mechanism
A cheap pixel-level segmentation of the raw screenshot proposes candidate regions. It uses connected components and edge boxes, which work well because rendered UIs have flat fills and sharp edges. Each candidate region $r$ gets a key vector by pooling high-resolution tiled features, as in Monkey. Monkey splits large images into 448-pixel tiles through a shared encoder with a global summary.
We add a dedicated cross-attention head to a decoder-only vision-language model. The decoder's current state is the query and the region keys are keys and values. This is the asymmetric routing that LXMERT and GLIP found necessary for fine-grained grounding. Here, though, it serves the output side, not the encoder. At step $t$ the next output unit $y_t$ is drawn from
$$P(y_t) = g^{\text{gen}}_t\, P_{\text{vocab}}(y_t) + \sum_{k} g^{k}_t \sum_{r} \alpha_{t,r}\, \mathbb{1}\left[M_k(r, I) = y_t\right]$$
The symbols are as follows. $P_{\text{vocab}}$ is the ordinary next-token distribution. $\alpha_{t,r}$ is the pointer's attention over candidate regions. $M_k$ is one of four measurement operators applied to region $r$ of the full-resolution image $I$: text transcription from a native-resolution zoomed crop, dominant fill or foreground color, pixel width or height, and position relative to the parent region. The gates $g^{\text{gen}}_t, g^{k}_t$ are softmax outputs that sum to one. A measured value such as #3a7bd5 or 14px is emitted as a single unit, and decoding then continues.
Supervision is free, because the training data is renderable HTML. A headless browser gives every DOM element's bounding box and computed style. For each literal in the target code we therefore know which operator should fire and which candidate region best overlaps the element that owns it.
The contrastive ingredient is where contrastive code representation learning enters, and it plays a supporting role rather than the headline. We add an InfoNCE loss, which pulls a matching pair of embeddings together and pushes mismatched ones apart. It aligns the decoder state at each literal-bearing code span with the key of the region that span renders to. The hard negatives are adjacent or look-alike regions: a sibling button of the same color, the next row of a table.
Two papers motivate this. ConCoder combines cross-entropy with InfoNCE over execution trajectories and found that plain cross-entropy absorbs failing code. ContraCode builds positive pairs from semantics-preserving compiler transformations. Both define equivalence through textual or interpreter execution. Here the "execution" is browser rendering, and the negatives are near misses in screen space. This targets the known weakness of contrastive code methods: without near-miss negatives, they cluster nearly correct outputs with correct ones.
Research Program
We would build the model on a 7B open vision-language backbone with native-resolution tiling. Training data is WebSight-style synthetic pages plus the VisDiffUI seed pages from VisRefiner, with render-derived pointer labels.
The analysis has three parts. The first is error attribution. We bin every element on Design2Code-HARD by rendered area and report per-bin text, color and position accuracy for all systems, to test whether gains concentrate on small elements as the bottleneck account predicts. The second is candidate-region recall. The segmentation step bounds the method: if it misses a region, the pointer cannot select it. We therefore measure what fraction of ground-truth DOM elements are covered at a given overlap threshold, and report results with the model's fallback to generation left on. The third is gate behavior. We check how often the model generates a value it should have measured, and the reverse. This follows the concern in Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That! that unchecked copying damages target structure. In our setting that damage would appear as copied absolute pixel widths where the reference uses relative or flex layout.
Baselines
The naive reference point is the same 7B backbone fine-tuned with plain cross-entropy on identical data, at the low input resolution the base model ships with. This establishes how much any intervention buys.
The resolution-only baseline is the same backbone with UReader-style native-resolution tiling, fine-tuned on the same data for the same number of steps. It is the most direct competitor, because it attacks the same bottleneck by feeding more pixels to the encoder rather than by copying. The proposal must beat it on Design2Code-HARD color similarity, position similarity and small-element text accuracy at equal training compute and a matched visual-token budget.
The strongest published methods are VisRefiner and UI2Code^N. VisRefiner trains on visual-difference repairs and then uses reinforcement learning with a reward based on CLIP embedding distance. It reports a Block-Match of 78.1 on Design2Code-HARD with a 7B model, against 56.6 for GPT-4o. It also reports a peak of 94.2 after five turns of filtered refinement. UI2Code^N uses pairwise visual preference optimization and reports 88.6% on Design2Code-HARD at 9B. It also reports Qwen2.5-VL-72B at 41.9% on the same subset.
The file does not state clearly whether these headline numbers are single-turn or which exact metric UI2Code^N's 88.6% refers to. We will therefore rerun VisRefiner's training recipe on our backbone and data, and report it at one turn and at five turns. The comparison that counts is a single-pass proposal against single-pass VisRefiner. Beating 78.1 Block-Match single-pass would be the target. A second claim is compositional: our model used as the first draft inside VisRefiner's loop should reach its plateau in fewer turns and oscillate less.
The pipeline baseline, called measure-then-prompt, takes the identical candidate regions and runs the identical measurement operators on them in advance. It serializes the results (region id, box, color, transcribed text) as text in the prompt. The model is fine-tuned to use them, with LargePiG-style copying over those prompt tokens. This is the ablation that isolates the key ingredient. It holds full-resolution measurement fixed and removes only the learned, context-conditioned cross-modal pointer.
Two further ablations complete the set. One removes the render-contrastive loss, keeping only cross-entropy pointer supervision. The other replaces the dedicated pointer head with training-free pointing read from the backbone's existing self-attention over visual tokens, in the manner of LargePiG.
Why not just…?
Why not just raise the input resolution with tiling and fine-tune? This is what UReader and Monkey do, and it clearly helps: UReader reports DocVQA rising from 22.0 to 65.4 with cropping. Tiling still forces the decoder to rebuild exact hex values and pixel offsets from compressed patch features. It also multiplies visual tokens, and the file notes that attention spreads thin over those tokens as sequences grow. The resolution-only baseline tests this directly. If it matches us on the small-element bins, the pointer is unnecessary and we will say so.
Why not just run OCR and a detector first and put the results in the prompt? This is the most serious threat, and it is exactly the measure-then-prompt baseline. Design2Code's text-augmented prompting helped with text content but did not recover layout or styling. That setup supplied only text, however, not measured colors and boxes, so it does not settle the question. Our pointer has two arguable advantages. First, a region costs one key vector instead of dozens of serialized tokens. Second, the choice of region is conditioned on which element the code is currently describing. If measure-then-prompt matches the full model, the honest conclusion is that the gain comes from measuring at full resolution, not from end-to-end pointing. That would still be a useful and publishable finding, and the pointer head should then be dropped.
Why not just iterate: render the draft, compare, and fix it? VisRefiner and UI2Code^N do this and report strong numbers. Both papers also report over-correction and late-turn oscillation, and VisRefiner's reward comes from CLIP at low resolution, which cannot see one-pixel errors. Refinement is complementary rather than competing. We test it both ways: our model against refinement at one turn, and our model as the first draft inside the refinement loop.
Why not just use supervised pointer labels and skip the contrastive loss? This may well be enough, since render-derived labels are dense and exact. The contrastive ablation decides it. We expect the loss to help mainly on pages with many look-alike siblings, where plain supervision can point at the wrong copy of a repeated component. If it does not help there, we drop it.
Proposed Experiments
The primary evaluation is Design2Code and Design2Code-HARD. We use their Block-Match, text, position, color and CLIP similarity metrics, add the area-binned element accuracy described above, and compare all systems at one generation pass. The single decisive check is the small-element bins on HARD. The full model should beat both the resolution-only baseline and measure-then-prompt on color and position accuracy for elements under a fixed small area threshold. We expect the gap on large elements to be small, because large elements survive downsampling. If the gain is instead spread evenly across element sizes, the resolution account is wrong and the improvement comes from something else, such as extra supervision.
VisDiffUI-Test supplies controlled, rule-based perturbations: color shifts, misalignments, reordering. We use it as a sensitivity probe. For each perturbation magnitude we ask whether the regenerated code reproduces the perturbed value exactly, which shows the smallest color or pixel shift each system can see. No reported result on Lacuna was found for any system's VisDiffUI-Test scores, so we report all systems ourselves.
For transfer beyond webpages, we evaluate on the reproduction level of the Chart2Code benchmark from From Charts to Code. Axis labels, tick values and series colors there are exactly the kind of literals the pointer targets. We report execution rate and the benchmark's rendered-fidelity score against the same baselines. That paper reports GPT-5.2 reaching a 96.04% execution rate on Level 2 editing but only a 33.03 fidelity score, a gap consistent with exact values being lost. We also report candidate-region recall and gate confusion alongside every headline number, so a reader can see how much of any shortfall comes from segmentation misses rather than from the pointer.
References
- Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering
- UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model
- VSA:Visual-Structural Alignment for UI-to-Code
- Get To The Point: Summarization with Pointer-Generator Networks
- Code Completion with Neural Attention and Pointer Networks
- LargePiG: Your Large Language Model is Secretly a Pointer Generator
- Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers
- Grounded Language-Image Pre-training
- Extracting the Essence and Discarding the Dross: Enhancing Code Generation with Contrastive Execution Feedback
- Contrastive Code Representation Learning
- VisRefiner: Learning from Visual Differences for Screenshot-to-Code Generation
- Pointer-Generator Networks for Low-Resource Machine Translation: Don’t Copy That!
- UI2Code$^{\text{N}}$: UI-to-Code Generation as Interactive Visual Optimization
- From Charts to Code: A Hierarchical Benchmark for Multimodal Models