Coding agents that build web pages now improve their output in a loop. They write HTML and CSS, render it in a headless browser, show the screenshot to a vision-language model (a model that reads images and text together) acting as a critic, and then revise the code based on its feedback. Many of these loops keep a revision only if the critic scores it higher than the previous best. That rule is a selection process. Whatever the critic systematically prefers, the loop will select for, round after round. Suppose the critic mildly prefers pages that show young, elite-credentialed people in a "candidate profile" card, or pages that look polished while their accessibility markup is broken. Then five rounds of keep-only-improvements should turn that mild preference into a strong property of the final page. Nobody has measured this.
We propose to measure it and to fix it. The measurement treats refinement depth as an amplifier. We quantify how demographic skew, accessibility loss and hidden code bloat grow from round 0 to round 5 in vision-critic loops, compared against loops without a visual critic. The fix borrows the central lesson of deepfake forensics: judges that rely on surface appearance stop working once generators learn to polish the surface, so a judge should check properties that must hold no matter how the surface looks. Concretely, a revision is accepted only if its score improvement survives a demographic swap of the page content and agrees with checks on the page's underlying structure. We call this an invariant acceptance rule.
Core Claim
In vision-guided refinement loops that use a keep-only-improvements rule, the loop's final demographic and accessibility skew is predicted by a quantity we can measure cheaply in one step: the critic's counterfactual preference gap. This is the average score difference the critic assigns to two renders that differ only in a sensitive attribute, or only in markup that changes nothing visible. We further claim that requiring score improvements to hold under paired counterfactual renders removes most of this amplification. It should do so while keeping most of the quality gain that makes these loops worthwhile.
Written out, if $s(x)$ is the critic's score for a rendered page $x$, and $x^{a}$ and $x^{b}$ are the same page with the sensitive attribute set to values $a$ and $b$, the one-step gap is
$$\Delta = \mathbb{E}\left[s(x^{a}) - s(x^{b})\right].$$
The invariant rule accepts a revision from page $x$ to page $x'$ only when
$$\min_{v \in V}\left[s(x'^{v}) - s(x^{v})\right] > 0,$$
where $V$ is the set of counterfactual variants: attribute swaps, plus a version of the render with decorative boilerplate masked out. We use this expression again when defining the ablations.
The claim is new because every piece of it has been studied only in isolation. Vision-Guided Iterative Refinement for Frontend Code Generation builds the critic-in-the-loop pipeline and shows visual feedback drives up to a 17.8% gain over three cycles. It evaluates layout and task completion only, and itself notes that its visual judges over-weight particular aesthetics. ReLook introduces the strict keep-only-improvements rule it calls Forced Optimization, and uses a multimodal critic as a reward during reinforcement learning. It rewards only visual fidelity and never asks what else that rule selects for. Both papers supply our method. We reuse their loops unchanged as the systems under study.
Why this intersection, and what each direction contributes
The bias direction motivates the proposal. FairCode shows that code models that avoid overt race and gender bias still write scoring logic that favors younger applicants and children of PhD holders, with test-case generation acting as a "bias magnet." It evaluates only one-shot Python. FairBelief finds that harmful associations sit at the top of the prediction distribution rather than in the tail. That matters here because a selection loop pushes outputs toward exactly those high-likelihood defaults. Investigating Subtler Biases in LLMs documents a halo effect: GPT-4 attributes positive traits to attractive people, high-GDP nationalities and elite institutions. A vision-language critic judging "quality" of a profile page is a natural channel for that effect. Uncovering Stereotypes in LLMs shows refusals drop as task complexity rises. Refinement adds complexity with every round, so we expect alignment to weaken across rounds.
The obsolescence direction supplies the lens and the fix. Celeb-DF showed that detectors relying on surface glitches fell from above 80% AUC on an older benchmark to 56.9% average once those glitches were removed. Exposing Deep Fakes Using Inconsistent Head Poses responded by checking a property that must stay consistent, the agreement between face and head geometry, rather than looking for glitches. Our screenshot critic is in the same position as a pixel-level detector: the generator is optimized directly against it, so its surface judgments are the first thing to be gamed. The invariant rule transfers the geometric-consistency idea to web pages. The score must stay consistent across content swaps, and the visible render must agree with the page's accessibility tree, the structured description of the page that screen readers use.
The boilerplate direction acts as a stress condition and a hidden-cost metric. SlopCodeBench found agent code accumulates verbosity about 7x faster than human code across iterations, but only for backend Python. A screenshot critic cannot see markup that does not render. Front-end loops should therefore accumulate invisible bloat at least as fast. We also test whether cookie banners, sticky navigation and ad containers in starter templates pull the critic's attention away from the content that matters. WebWalker and LiveWeb-IE document this kind of distraction for text agents and extractors, not for critics.
The closest multi-direction work is AccessGuru. It combines rule-based DOM checks from the Axe-Playwright accessibility engine with GPT-4o screenshot inspection to repair existing HTML, and reports an 84% average violation-score decrease. It repairs isolated snippets toward a fixed standard. It does not study how an open-ended quality critic drifts a page over rounds, and it does not address demographic content. We use its violation taxonomy (syntactic, layout, semantic) as our accessibility metric.
Research Program
The first component is a task suite, FairFront. It turns FairCode's three domains (hiring over 540 occupations, admissions over 320 majors, medical treatment over 95 treatments) into front-end generation tasks. Examples include a recruiter dashboard that lists and sorts candidates, an admissions landing page with sample student testimonials, and a clinic triage intake form. Each task leaves demographic content open to the generator: who appears in placeholder profiles, which fields a form requires, the default sort order, and which applicants get highlighted. For boilerplate stress, each task comes in two versions: a clean starter template and one carrying realistic cookie consent overlays, sticky navigation and ad slots. For quality, we add the ArtifactsBench-Lite tasks used in ReLook and the WebDev Arena test split of 600 tasks used by the critic-in-the-loop paper, so that fixes are also judged on standard quality benchmarks.
The second component is an amplification audit. We run each loop for five rounds, since gains are reported to plateau after three to five (Yan et al.), and log four quantities at every round. Demographic skew is measured with a FairScore-style preference entropy over the attributes that appear in generated content and sort logic, using FairCode's six categories plus the beauty, age and institution axes from the subtle-bias benchmark. Accessibility is the axe violation count plus GPT-4o semantic violations under the AccessGuru taxonomy. Invisible bloat is the ratio of DOM nodes and CSS rules to pixels that actually change on screen. Quality is the ArtifactsBench judge score. The amplification ratio for each metric is its value at round $t$ divided by its value at round 0.
The third component is a predictor test. For each critic (Qwen2.5-VL as in ReLook, and a stronger proprietary critic), we measure $\Delta$ once on fixed pairs of renders that differ only in the attribute. We then test whether $\Delta$, together with the spread of the critic's scores, predicts the amplification ratio across critics, attributes and acceptance rules. The intuition comes from selection on a noisy score: under keep-only-improvements, drift along a direction the critic prefers grows with the number of accepted rounds times the ratio of $\Delta$ to that spread. We state this only as a trend to be checked, not a derived law.
The fourth component is the invariant critic defined by the acceptance rule above. The variant set $V$ contains the attribute-swapped renders and a boilerplate-masked render. It is paired with a structural consistency check: an edit is rejected if it improves the screenshot score while adding axe violations, or while growing invisible bloat beyond a set tolerance.
Baselines
The naive reference is single-shot generation, round 0, with no refinement. Every amplification ratio is defined relative to it, and it anchors FairCode's one-shot findings in the front-end setting.
The second reference is text-only refinement, the "Refine w/o Critic" condition of Vision-Guided Iterative Refinement, which gained only 1.5%. It separates what iteration alone does from what the visual critic's selection pressure does. If skew grows just as much without the critic, our mechanism is wrong.
The strongest existing methods are the two published vision-critic loops, run as their papers describe. The first is the critic-in-the-loop pipeline, with a separate visual critic, code critic and improver. The second is ReLook's Generate-Diagnose-Refine loop with Forced Optimization and its zero-reward rule for pages that fail to render. The strongest reported number on our quality side is ReLook raising Qwen2.5-7B from 21.59% to 27.88% on ArtifactsBench-Lite. For the bias side, we found no reported result on Lacuna for demographic or accessibility skew in any iterative front-end refinement loop.
For accessibility repair specifically, the strongest competitor is AccessGuruCorrect (AccessGuru), run after the final round as a post-hoc repair pass.
For fairness of comparison, all loops share the same generator (Qwen2.5-7B and one proprietary model), the same starter templates, the same five-round cap, and the same number of critic calls. The invariant critic needs roughly $|V|+1$ critic calls per round. Baselines get the same budget as best-of-$(|V|+1)$ sampling of candidate revisions under their own acceptance rule, so any gain cannot come from extra compute. Critic prompts are tuned for each method on the same 200-task validation split.
Beating the baselines means two things. First, the invariant loop's amplification ratio for demographic skew and for accessibility violations stays at or below 1.1, meaning no more than a 10% rise over round 0, while the published loops exceed that. Second, it retains at least 80% of ReLook's quality gain over round 0 on ArtifactsBench-Lite, measured with the same judge.
Two ablations isolate the key ingredients. The first keeps only the attribute swaps in $V$, dropping the boilerplate mask and the structural check. The second keeps only the structural check. Together they show whether counterfactual pairing or structure agreement does the work for each metric.
Why not just…?
Why not just tell the generator and critic to be fair and accessible? This is the cheapest option, and we run it: every baseline loop gets a matched condition in which a fairness and accessibility instruction is added to both generator and critic prompts. The evidence against it as a full answer is fairly strong. SlopCodeBench found quality prompts reduced early bloat but did not stop later degradation and slightly hurt correctness. FairCode found prompting was not a "silver bullet" for age, income or parental-education bias. Our prediction is that prompting shifts the round-0 level down but leaves the slope across rounds intact, because the critic's residual gap $\Delta$ is still being selected for.
Why not just add an accessibility linter to the reward? This is a real threat on the accessibility axis. A reward penalizing axe violations may match our structural check, and we include exactly that as a baseline. It cannot address semantic violations, which AccessGuru reports rule-based tools miss. It also cannot address demographic content, which no linter inspects. If the linter-augmented loop matches the invariant critic on accessibility, we will say so. The contribution then narrows to demographic amplification and to the predictor result.
Why not just stop refining after one or two rounds? If amplification grows with depth, early stopping trades quality for fairness. Given the reported three-to-five-round plateau, it might be an acceptable trade. We report the full quality-versus-skew curve for every stopping round. If round-2 stopping reaches the same point on that curve as the invariant critic at round 5, the invariant rule is not worth its cost, and we will conclude that.
Why not just use a stronger or less biased critic? A stronger critic may have a smaller $\Delta$. The predictor test addresses this directly. If amplification tracks $\Delta$, a better critic reduces amplification only as much as it reduces $\Delta$, and $\Delta$ is cheap to measure before deployment. If a stronger critic has near-zero $\Delta$ on every attribute, the invariant rule is redundant for that critic, which is itself a useful finding.
Decisive Checks
Four results would settle the proposal. The first is existence: over five rounds, vision-critic loops must raise demographic skew and invisible bloat above round 0 by clearly more than text-only refinement does, with confidence intervals from paired resampling over tasks. If they do not, the core claim fails and the paper becomes a negative result about critic selection pressure.
The second is prediction: across at least two critics and nine attributes, the one-step gap $\Delta$ must rank-correlate with the amplification ratio. If it does not, the selection explanation is wrong even if amplification exists.
The third is repair: the invariant rule must meet the thresholds stated in the Baselines section under equal critic budget. The ablations must also show which part of $V$ handles which metric.
The fourth is boilerplate: on templates carrying heavy boilerplate, the critic's feedback should shift toward peripheral elements. Measured by where its diagnostic comments point on the page, this should lower quality gains relative to clean templates, and the boilerplate-masked variant should recover that loss. A null here is informative. It would mean visual critics, unlike the text-DOM agents in WebWalker and LiveWeb-IE, already ignore this clutter.
References
- Vision-Guided Iterative Refinement for Frontend Code Generation
- ReLook: Vision-Grounded RL with a Multimodal LLM Critic for Agentic Web Coding
- FairCode: Evaluating Social Bias of LLMs in Code Generation
- FairBelief - Assessing Harmful Beliefs in Language Models
- Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models
- Uncovering Stereotypes in Large Language Models: A Task Complexity-based Approach
- Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics
- Exposing Deep Fakes Using Inconsistent Head Poses
- SlopCodeBench: Measuring Code Erosion Under Iterative Specification Refinement
- WebWalker: Benchmarking LLMs in Web Traversal
- LiveWeb-IE: A Benchmark For Online Web Information Extraction
- AccessGuru: Leveraging LLMs to Detect and Correct Web Accessibility Violations in HTML Code
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation