Prompt-Compiled Layout Checks as Cheap Rewards for Visual Code RL

Models that write webpages or plotting code are now trained with reinforcement learning (RL): the model writes code, a browser or plotting library draws it, and a large vision-language model (a model that looks at an image and answers questions about it) grades the picture. The grade becomes the reward. This works, but it is slow and noisy. Every sampled program in every training step needs a forward pass through a judge as large as GPT-5 or a 72B Qwen model, and those judges give poorly calibrated scores. We propose to move the expensive language understanding out of the training loop. Before training, each natural-language request is translated once into a small program of checkable layout constraints, such as "the legend does not overlap any data," "there are three cards in one row," or "the title sits above the chart and its text fits inside its box." During RL, each rollout is rendered as usual, and the rendering engine's own geometry is read out: the bounding box, text, and color of every page element or plot artist. The constraints are scored on that geometry with no neural judge in the loop.

The bet is that most of what a vision-language judge contributes to training on open-ended prompts is geometric and structural, and that a judge is needed to understand the request, not to look at every candidate. If so, judge calls drop from one per rollout to one per prompt. The reward also becomes localized, because a violated constraint names specific elements and therefore specific lines of code. That localization gives a principled stopping rule for iterative refinement: stop when the constraint set stops changing. We test the bet against judge-in-the-loop RL at matched compute on interactive web artifacts and on text-to-chart generation.

Core Claim

For open-ended visual code generation, a reward built from prompt-specific constraints compiled offline and checked on the rendered layout tree can match reward from a per-rollout multimodal judge. It should do so at a small fraction of the judge compute and win outright at equal wall-clock time. A light, periodic judge audit is enough to catch the reward hacking that partial constraints invite.

Written out, the reward for code $y$ on prompt $x$ is

$$R(x,y) = \mathbb{1}[\text{renders}(y)] \cdot \frac{\sum_{c \in C(x)} w_c\, s_c(L(y))}{\sum_{c \in C(x)} w_c},$$

where $C(x)$ is the constraint set compiled from prompt $x$ once, before training. $L(y)$ is the layout tree extracted from the rendered output: element boxes, text, and computed styles from the browser, or artist extents from Matplotlib. $s_c \in [0,1]$ is a deterministic soft score for constraint $c$, and $w_c$ is its weight. The render gate follows the zero-reward rule of ReLook.

Why this gap is real

Four strands of prior work motivate the proposal. RL-Text2Vis aligns text queries, plotting code, and charts with group relative policy optimization, or GRPO: a variant of policy gradient that scores each sample against the other samples drawn for the same prompt, so no separate value network is needed. Its visual reward still queries Qwen2.5-VL for every rollout, and training a 14B policy took 50 hours on six H100s. ReLook removes the multimodal critic at test time, which cuts latency from 123.04 to 18.03 seconds per query, but it keeps the critic in every training step. Code Aesthetics with Agentic Reward Feedback stacks a GPT-5 screenshot judge and a clicking GUI agent into the reward. That adds heavy latency and navigation noise, and it reports a drop in general coding ability. UI2Code^N shows that absolute judge scores are too noisy to use directly and replaces them with pairwise tournaments among candidates. That fixes calibration but multiplies judge calls. All four papers attack the judge's noise or the test-time cost. None of them removes the per-rollout judge from training.

Two papers supply the method and also constrain it. ViCT already skipped the browser during RL by training a critic that predicts visual IoU from code, which shows that rendering-free or judge-free rewards can work. Its critic, however, compares against ground-truth code and cannot handle open-ended prompts. We keep rendering, which is cheap, and drop the reference, which does not exist for open-ended prompts. Breaking the SFT Plateau shows that cheap rule-based rewards carry much of the RL gain in chart-to-code, since its first training stage uses only those rules. Its rules check data values, chart type, and layout against a reference chart. We replace the reference with constraints derived from the request itself.

On the semantic gap side, WebCode2M and ViCT show that structure-aware measures track visual quality far better than token overlap. ViCT reports that htmlBLEU correlates at 0.764 with visual similarity, against 0.329 for standard BLEU. MatPlotAgent catalogs the defects that plain execution checks miss: overlapping labels, clipped containers, and unreadable axes. These defects are geometric, and a layout tree can detect them. The part geometry cannot express is intent, which vague prompts such as "make it look professional" leave underspecified, as UI2Code^N notes. We handle intent with a large model once per prompt rather than once per sample.

Research Program

The first component is a constraint language and a compiler. The language is a small typed vocabulary over layout trees. It covers presence and count of element kinds, relative position (above, left-of, inside, aligned), non-overlap, text containment and overflow, minimum readable font size and color contrast, data-binding checks for charts (the values plotted match the table within a tolerance, following the ±5% rule in MSRL), and, for interactive pages, pre- and post-conditions around scripted interactions in the style of the three-snapshot protocol of ArtifactsBench. A frontier model compiles each training prompt into a weighted constraint set. It is prompted with the prompt text and a few reference renders sampled from a strong model, and asked for constraints that any acceptable output must satisfy. Compilation cost is paid once and reported in full.

The second component is layout extraction. For HTML, Playwright reads computed boxes and styles for the DOM. For Matplotlib, the renderer reports the extent of each artist. Both are measured for per-rollout cost.

The third component is training. GRPO runs on the constraint reward. A judge audit scores a random 5–10% of rollouts with a multimodal judge. The audit does not drive gradients directly. It serves as a tripwire: when constraint reward and audit score diverge beyond a threshold on a prompt, that prompt's constraint set is flagged for recompilation.

The fourth component is test-time refinement. The policy runs a self-edit loop like ReLook's. At each step it is shown the list of violated constraints and the elements responsible, instead of a free-text critique. It stops when the violated set is empty or when it repeats. This is aimed directly at the oscillation reported in UI2Code^N.

Baselines

The naive reference points are the base policy with no RL, and the same base policy trained with GRPO on the execution/render gate alone. The second shows how much any visual signal adds beyond "it runs."

The strongest competitor is GRPO with a per-rollout multimodal judge, reimplemented to follow RL-Text2Vis for charts and ReLook for web. It is the strongest competitor because it is exactly the design we claim is wasteful, and it is the current best open recipe on these benchmarks. For reference, ReLook-7B reports 27.88% on ArtifactsBench-Lite against 21.59% for base Qwen2.5-7B. RL-Text2Vis-14B reports a 29% final pass rate on Text2Vis, up from 14% zero-shot and against 30% for GPT-4o, along with 97% execution and 4.10/5 readability. To keep the comparison fair, both arms start from the same base model, use the same training prompts, group size, and learning-rate sweep, and run the same test-time self-edit budget. Comparisons are reported at matched optimizer steps and at matched total GPU-hours, with judge compute, compilation, and audit costs all counted. Beating it means matching its benchmark scores within noise at matched steps while using at least five times less reward compute, or exceeding it at matched GPU-hours.

The third baseline is the two-stage curriculum of MSRL: cheap rules first, then the judge on a curated subset. It is the existing answer to the compute problem, so the constraint reward must beat it at equal compute.

The ablations that isolate the key ingredient are three. The first replaces prompt-compiled constraints with generic, prompt-agnostic layout lints (overlap, overflow, contrast), which tests whether per-prompt compilation matters. The second removes the judge audit, which measures reward hacking. The third substitutes a small distilled judge (a 7B vision-language model fine-tuned on large-judge scores) for the constraint reward.

Why not just…?

Why not just distill the big judge into a small reward model? This is the most serious threat, and it is run as a full arm at the same compute. A small judge still costs a vision forward pass per rollout, still gives unlocalized scalar scores, and inherits the calibration problem UI2Code^N documents, possibly in worse form. No paper in our file reports a distilled visual code judge, so the question is open. If the distilled judge matches constraints at equal cost, the claim narrows to localization and refinement stopping, and we will say so.

Why not just use MSRL's rules-then-judge curriculum? Its rules need a reference chart. For open-ended prompts its second stage falls back to the per-rollout judge, so the savings disappear exactly where the prompts are open-ended. The matched-GPU-hour comparison settles whether compiled constraints extend rule-based rewards into that regime.

Why not just use generic lint rules without compiling each prompt? Generic lints catch overlap and overflow but not "three cards per row" or "bars sorted descending," and those intent failures are what the semantic gap is about. The generic-lint ablation measures this directly. If generic lints recover most of the gain, the compiler is unnecessary, and the finding would still matter for compute.

Why not just skip RL and self-refine with a frontier model at test time? ReLook measures such loops at about 123 seconds per query. ChartMimic reports open models that often fail to execute at all, which leaves the critic nothing to look at. We include frozen-model self-refinement with constraint feedback as a no-training reference.

Proposed Experiments

Web experiments train on webpage instructions from AesCode-358K and evaluate on ArtifactsBench-Lite and OpenDesign. On OpenDesign, AesCoder-4B reports 81.92 and GPT-4o 48.08. Chart experiments train on Text2Vis and evaluate in-domain on Text2Vis, and out of domain on PandasPlotBench, VisEval, and NVBench. For VisEval and NVBench, no reported numeric result was found on Lacuna, so we report our own reruns of all arms. Every arm is also evaluated on LiveCodeBench to track the alignment tax that Code Aesthetics reports. No numeric LiveCodeBench drop is reported in our file, so we measure it for every arm from the same base.

Final scoring always uses judges held out from training, namely the benchmarks' own dual-judge protocols, plus a 200-item human preference study, so that the constraint arm cannot win by grading itself.

Four checks decide the outcome. The first is the quality-versus-GPU-hour curve of the constraint arm against the per-rollout judge arm. The claim fails if the judge arm dominates at every compute level. The second is compiler validity: on 300 human-annotated prompts, we measure the precision of compiled constraints and their recall of the requirements humans list. The third is hacking: we track how audit-judge score and constraint reward diverge over training, with and without the audit. The fourth is oscillation: on UIPolish-bench, we count refinement steps that change the code without changing the violated-constraint set, comparing constraint-gated stopping against ReLook-style score-gated acceptance. If constraint rewards match the judge at a fraction of the cost and cut wasted refinement steps, that suggests judges belong in training at compile time and in audits, not in every rollout.