TL;DR
ProductWebBench is a new benchmark for frontend AI agents that shifts the focus from simple task completion to "website continuity"—the ability to add features without breaking existing functionality. The authors find that even when top-tier models like GPT-4 and Claude successfully implement a requested change, nearly 30% of the time they leave the rest of the site in a broken state.
What Problem Does This Paper Address?
In traditional software engineering, building a feature is only half the battle; the other half is ensuring you haven't introduced regressions. Most current AI coding benchmarks (like SWE-bench or Design2Code) focus on whether an agent can build a specific target from scratch or fix a isolated bug.
However, real-world frontend development happens on "living" repositories where changes to a shared state or a CSS file can have unintended side effects across multiple pages. The authors identify a "continuity gap": a model might pass a local DOM assertion for a new shopping cart button while simultaneously breaking the mobile layout or causing console errors on the checkout page. ProductWebBench is designed to catch these "silent" failures that occur beyond the immediate scope of the requested task.
Methods
The authors introduced several novel mechanisms to evaluate continuity:
- Staged Requirements & Cumulative Gates: Instead of giving the agent a massive prompt, a simulated user reveals requirements one stage at a time. Crucially, once a model passes a check for an earlier stage, that check remains "armed" for the rest of the run. If a later edit breaks an earlier success, it's flagged as a regression.
- Provenance-Exact Regression: The benchmark tracks regressions through two channels:
- Replay: Re-verifying states the agent itself previously cleared.
- Untouched-Repository Channel: Running the same suite of checks on the original, unmodified repository to see if the agent broke something that was working perfectly before it started.
- Website Continuity Score (WCS): A graded metric that balances progress (stages cleared) with penalties for regressions.
- Admission Controls: To ensure the benchmark is fair, every task must pass a "reference implementation" (a human-written correct solution) and fail "plausible-wrong" solutions before being included.
Datasets
The authors introduced the ProductWebBench dataset, which consists of:
- 400 tasks in total, split into 320 "Change" tasks (modifying existing code) and 80 "Build" tasks (long-horizon construction).
- 8 framework families, including React, Vue, Svelte, Astro, and even static PHP/HTML sites.
- 8,737 evaluation runs across 13 different frontier and open-source models.
Key Insights and Ideas
- Completion $\neq$ Continuity: Passing the "target" check does not mean the website is healthy. There is a massive disconnect between an agent's ability to fulfill a request and its ability to preserve the system it lands in.
- Interaction is a Blind Spot: Static analysis (checking the DOM) is insufficient. 29.5% of runs that passed static DOM requirements failed when the benchmark actually tried to interact with the elements (e.g., clicking a button).
- Scaffolds Matter: The "agentic" wrapper (how the model is prompted and how it interacts with the file system) can move the needle more than the underlying LLM backbone. Swapping a scaffold under a fixed model changed scores by up to 10 points.
- Prompting for Preservation: Simply adding a "do not break unrelated behavior" clause to the prompt significantly improved continuity scores without hurting task completion.
Key Results
The study evaluated 13 models, including Claude-Opus, GPT-4 (referred to as GPT-5.5 in the paper's schematic tables), and various Qwen and DeepSeek variants.
- The Continuity Gap: Of the "Change" runs that successfully passed every requested-work check, 28.2% still failed a continuity check (like a blank route or console error).
- Regression Rates: 10.8% of successful runs pushed a browser state they had already passed back into a failing state.
- Incumbent Damage: 18.3% of runs were caught breaking a state that was verified as working in the original, untouched repository.
- Model Performance: While Claude-Opus and GPT-4 led the pack, their absolute "whole-task" success rates (clearing every stage without any regressions) were surprisingly low, ranging from only 1.25% to 3.50%.
Limitations and Future Work
- Strictness: The "Whole-task pass" metric is intentionally very difficult, which results in low absolute numbers that can be hard to use for fine-grained model ranking.
- Environment Sensitivity: Some continuity checks (like console errors or layout overflows) can be sensitive to browser versions and rendering speeds, though the authors implemented "Admission Gates" to mitigate this.
- Future Directions: The authors suggest expanding the benchmark to include more complex multi-role interactions (e.g., a PM and a Designer giving conflicting feedback) and exploring how agents can better "self-heal" when they detect a regression.
Summary updated