Keeping the Screenshot in the Loop: Native Render-Tool Training Against Forgetting

A model that writes web pages usually cannot see what it wrote. One fix runs a separate pipeline: render the page in a headless browser, show the screenshot to a vision model acting as critic, and feed the critique back to the code writer. A second fix fine-tunes the writer to produce the improved page in one shot. This proposal studies a third option and asks what it costs. We teach a vision-language model, meaning a language model that also reads images, to call a render function itself. The tool returns a screenshot, and the model edits its own code over several turns. The training data are written as interleaved documents in which code, tool calls, screenshots and edits alternate, the same image–text format such models already see in pre-training.

The central question is not only whether this improves frontend code. It is also whether this way of specializing damages the model's other abilities less than the alternatives do. Those abilities include general function calling, general visual question answering and text reasoning. The concern is real. Heavy specialization of an open prover on formal math dropped its general tool-calling accuracy from 89.4% to about 4.6% (Awakening the Sleeping Agent). The leading visual-refinement work distilled its loop into a single-pass model without ever measuring what that model forgot (Vision-Guided Iterative Refinement for Frontend Code Generation).

Core Claim

We hypothesize the following. Training a model to use a render observation, meaning reading a screenshot and making a local edit, moves the model's output distribution less than training it to emit the final refined program in one pass. It should therefore buy more target-task gain per unit of general-ability loss. The intuition is about the targets. Single-pass distillation asks the model to absorb, into its first guess, everything a three-round critic loop discovered, which is a large shift in what it writes from scratch. Tool-loop training mostly teaches a new conditional behavior: given this screenshot, change these lines. That behavior is learned on sequences whose shape, alternating images and text, already matches pre-training. VILA is the motivating precedent: it found that interleaved image–text data preserves text abilities better than isolated image–caption pairs because it matches the text distribution the model already knows.

We summarize each training recipe by a forgetting-per-gain ratio:

$$\rho = \frac{G_{\text{base}} - G_{\text{tuned}}}{T_{\text{tuned}} - T_{\text{base}}}$$

Here $G$ is a fixed average of general benchmarks, $T$ is the target frontend score at a fixed inference budget, and the subscripts denote the base and fine-tuned checkpoints. The claim is that native render-tool training achieves a lower $\rho$ than single-pass distillation at matched data and compute. It should also keep $\rho$ low when general replay data is added on top, rather than merely matching what replay alone achieves.

Research Program

The base model is an open vision-language model pre-trained on interleaved web data with an unfrozen language backbone, of the kind described by VILA and BLIP-3. We need this kind of model so that screenshots in the context are a familiar input rather than a new modality.

Trajectories come from the same 1,200 WebDev Arena training tasks used by the CITL work. We run the CITL critic loop as a teacher and then rewrite each run as a single interleaved sequence. The sequence holds the task, the first draft, a structured render(html) call, the returned screenshot as an image, the model's own short critique, and a diff-style edit, repeated until the score stops rising. The external critics are thus folded into the model's own reasoning between tool calls.

Two additions address known failure modes. First, the Awakening the Sleeping Agent study found that reactivated models become "trigger-happy" when they never see examples where no tool is needed. We therefore include trajectories where the first draft is already good and the correct action is to stop without rendering. Second, refinement gains plateau after a few rounds (Yan et al.). Trajectories are therefore cut at the teacher's last improving step, so the model learns when to stop editing.

Our analysis goes beyond scores. We measure, on held-out general prompts, the KL divergence between the tuned and base models' output distributions, and the relative weight change $|\Delta W / W|$ per layer, following the drift measure of RECT. These test whether lower forgetting actually comes with smaller movement of the model's outputs and weights, which is the mechanism we propose. The tool harness is fixed across all conditions and token use is reported alongside pass rates. The Scaffold Effect in Coding Agents shows that changing the harness can change token cost about 40x while pass rates move only 0–8 points.

Baselines

The naive reference point is the untrained base model generating a page in one pass, plus the same base model prompted to call render under our harness with no training. The prompted condition tells us whether training is needed at all, or whether the base model can already use screenshot observations.

The strongest existing method for the target task is the external Critic-in-the-Loop pipeline of Vision-Guided Iterative Refinement, which uses a separate visual critic, code critic and improver. That paper reports up to a 17.8% improvement over three refinement cycles for Distill-Qwen-14B on WebDev Arena, against 1.5% for refinement without a critic. We will rerun it with our base model as the improver, on the same 600 test tasks and with the same three-cycle budget. We count the total tokens and model calls of all three roles. Beating it means matching its WebDev Arena gain at equal or lower total inference tokens using one model instead of three. Coming within a small margin at a fraction of the cost would still be a useful but weaker result, and we would report it as that.

The strongest existing method on the forgetting question is the same paper's LoRA distillation into single-pass generation, which captured 24.7% of the best CITL gains. We retrain it on the identical 1,200 tasks with the same base model, the same number of gradient steps and the same hyperparameter search budget, using both LoRA and full fine-tuning. We then measure the general benchmarks that paper never measured. Beating it means a lower $\rho$ at equal or higher target score.

Two further baselines address forgetting directly. The first is general data re-blending in the style of VILA, applied at the same mixing ratio to every training condition. The second is post-hoc reactivation in the style of Awakening the Sleeping Agent, which restored BFCL accuracy to 83.8% with 100 traces. We apply it after distillation and check both BFCL and irrelevance detection.

Two ablations isolate our key ingredients. One keeps the multi-turn tool trajectories but replaces each screenshot with a text DOM summary, which tests whether the visual observation matters. The other removes the no-render examples, which tests the irrelevance fix.

Why not just…?

Why not just keep the external critic pipeline and never fine-tune? Frozen weights cannot forget, so this is a serious alternative. Its costs are three models per query and an outer orchestration script rather than a skill the model has. If our evaluation shows it beats native training at equal total tokens, then native training is not worth it for quality alone. We would say so, and the remaining argument would be deployment simplicity. The matched-token comparison in Baselines decides this.

Why not just distill the loop into a one-shot generator? It captured only 24.7% of the loop's best gains in the original work (Vision-Guided Iterative Refinement), and its forgetting cost is unknown. Our head-to-head $\rho$ comparison is exactly the experiment that work omitted. If distillation shows equal $\rho$, our core claim fails.

Why not just fine-tune however you like and then reactivate with a hundred traces? Awakening the Sleeping Agent shows tool use is suppressed rather than erased, so this cheap repair might close the gap. That evidence is text-only and Lean-specific, and the repaired model over-calls tools. We run the repair on every condition and report BFCL irrelevance detection separately, so a repair that restores aggregate accuracy while breaking abstention will be visible.

Why not just add replay data? Replay is the most likely way our claim could fail. If replay brings distillation's $\rho$ down to ours, then the trajectory format is irrelevant and mixture is what matters. We therefore report every condition with and without identical replay. The claim survives only if native tool training stays lower than distillation under replay as well.

Proposed Experiments

Target evaluation uses three benchmarks. The first is the WebDev Arena test split with a multi-dimensional VLM judge, following CITL. The second is Design2Code and its 80-case HARD subset; GPT-4o reports Block-Match 93.0 on the full set and 56.6 on HARD, which serves as a frontier reference rather than a target for an open model. The third is ArtifactsBench, which tests interactive artifacts with before/during/after screenshots; no reported per-model result was found on Lacuna for it. We plot score per refinement round to see where gains plateau.

General evaluation uses three sets of benchmarks. The first is the full BFCL, with the irrelevance and multi-turn categories reported separately; the 89.4% base Qwen3-32B figure is the only reported anchor and comes from a different model. The second is TextVQA and VisWiz, following VILA. The third is a held-out text reasoning suite.

The study turns on three checks. The first is whether native render-tool training yields lower $\rho$ than single-pass distillation, with and without replay. The second is whether the screenshot-to-DOM ablation erases the advantage, which would show the visual observation, not just the multi-turn structure, is doing the work. The third is whether lower $\rho$ comes with lower KL divergence and smaller relative weight drift. If it does not, the proposed mechanism is wrong even where the scores favor us. We run three seeds per condition and report confidence intervals, because $\rho$ divides one small difference by another and is noisy.