ProductWebBench: Evaluating Website Continuity Beyond Task Completion

Jiajun Wu, Jian Yang, Yaxin Du, Shukai Liu, Linzheng Chai, Tuney Zheng, Wei Zhang, Haitao Yang, Yan Bai, Yihang Lou, Xianglong Liu, Bryan DaiarXiv 2026 · arXiv

TL;DR

ProductWebBench is a new benchmark for frontend AI agents that shifts the focus from simple task completion to "website continuity"—the ability to add features without breaking existing functionality. The authors find that even when top-tier models like GPT-4 and Claude successfully implement a requested change, nearly 30% of the time they leave the rest of the site in a broken state.

What Problem Does This Paper Address?

In traditional software engineering, building a feature is only half the battle; the other half is ensuring you haven't introduced regressions. Most current AI coding benchmarks (like SWE-bench or Design2Code) focus on whether an agent can build a specific target from scratch or fix a isolated bug.

However, real-world frontend development happens on "living" repositories where changes to a shared state or a CSS file can have unintended side effects across multiple pages. The authors identify a "continuity gap": a model might pass a local DOM assertion for a new shopping cart button while simultaneously breaking the mobile layout or causing console errors on the checkout page. ProductWebBench is designed to catch these "silent" failures that occur beyond the immediate scope of the requested task.

Figure 1: Building from scratch vs. changing a living website.

Methods

The authors introduced several novel mechanisms to evaluate continuity:

  • Staged Requirements & Cumulative Gates: Instead of giving the agent a massive prompt, a simulated user reveals requirements one stage at a time. Crucially, once a model passes a check for an earlier stage, that check remains "armed" for the rest of the run. If a later edit breaks an earlier success, it's flagged as a regression.
  • Provenance-Exact Regression: The benchmark tracks regressions through two channels:
  • Replay: Re-verifying states the agent itself previously cleared.
  • Untouched-Repository Channel: Running the same suite of checks on the original, unmodified repository to see if the agent broke something that was working perfectly before it started.
  • Website Continuity Score (WCS): A graded metric that balances progress (stages cleared) with penalties for regressions.
  • Admission Controls: To ensure the benchmark is fair, every task must pass a "reference implementation" (a human-written correct solution) and fail "plausible-wrong" solutions before being included.
Figure 3: Online multi-turn evaluation. PM/QA/designer are alternative personas, not concurrent. The harness checks each milestone and replays every prior gate; inset values are schematic.

Datasets

The authors introduced the ProductWebBench dataset, which consists of:

  • 400 tasks in total, split into 320 "Change" tasks (modifying existing code) and 80 "Build" tasks (long-horizon construction).
  • 8 framework families, including React, Vue, Svelte, Astro, and even static PHP/HTML sites.
  • 8,737 evaluation runs across 13 different frontier and open-source models.

Key Insights and Ideas

  • Completion $\neq$ Continuity: Passing the "target" check does not mean the website is healthy. There is a massive disconnect between an agent's ability to fulfill a request and its ability to preserve the system it lands in.
  • Interaction is a Blind Spot: Static analysis (checking the DOM) is insufficient. 29.5% of runs that passed static DOM requirements failed when the benchmark actually tried to interact with the elements (e.g., clicking a button).
  • Scaffolds Matter: The "agentic" wrapper (how the model is prompted and how it interacts with the file system) can move the needle more than the underlying LLM backbone. Swapping a scaffold under a fixed model changed scores by up to 10 points.
  • Prompting for Preservation: Simply adding a "do not break unrelated behavior" clause to the prompt significantly improved continuity scores without hurting task completion.

Key Results

The study evaluated 13 models, including Claude-Opus, GPT-4 (referred to as GPT-5.5 in the paper's schematic tables), and various Qwen and DeepSeek variants.

  • The Continuity Gap: Of the "Change" runs that successfully passed every requested-work check, 28.2% still failed a continuity check (like a blank route or console error).
  • Regression Rates: 10.8% of successful runs pushed a browser state they had already passed back into a failing state.
  • Incumbent Damage: 18.3% of runs were caught breaking a state that was verified as working in the original, untouched repository.
  • Model Performance: While Claude-Opus and GPT-4 led the pack, their absolute "whole-task" success rates (clearing every stage without any regressions) were surprisingly low, ranging from only 1.25% to 3.50%.
Figure 4: Post-completion failure rates across 13 models.

Limitations and Future Work

  • Strictness: The "Whole-task pass" metric is intentionally very difficult, which results in low absolute numbers that can be hard to use for fine-grained model ranking.
  • Environment Sensitivity: Some continuity checks (like console errors or layout overflows) can be sensitive to browser versions and rendering speeds, though the authors implemented "Admission Gates" to mitigate this.
  • Future Directions: The authors suggest expanding the benchmark to include more complex multi-role interactions (e.g., a PM and a Designer giving conflicting feedback) and exploring how agents can better "self-heal" when they detect a regression.

Summary updated

Abstract

Frontend coding agents are usually graded on one question: did the requested target get built. Real engineering changes carry a second obligation, because the rest of the site has to keep working. We call that second obligation website continuity, and introduce ProductWebBench, a benchmark that scores both on living frontend repositories: 320 Change tasks over existing code and 80 long-horizon Build tasks. Every task freezes a repository snapshot, staged requirements, browser evidence and deterministic checks before evaluation, and enters only once a reference passes every check and plausible-wrong ones fail. A simulated user reveals the requirements one stage at a time, and every check cleared earlier stays armed for the rest of the run. Across 8,737 runs from 13 models the two come apart: of the Change runs passing every requested-work check, 28.2% still fail a continuity check -- a blank route, horizontal overflow, console errors -- and 10.8% push a browser state they had already passed back to failing. Because the same checks also run on the untouched repository, each regression carries exact provenance, not a post-hoc label: on the 586 runs where both readings exist, 18.3% are caught breaking a verified state. Interaction is a further blind spot, failing in 29.5% of runs whose static DOM already passes. ProductWebBench therefore scores a patch against the living system it lands in, not the target alone.
BibTeX
@misc{wu2026productwebbench,
  author = {Jiajun Wu and Jian Yang and Yaxin Du and Shukai Liu and Linzheng Chai and Tuney Zheng and Wei Zhang and Haitao Yang and Yan Bai and Yihang Lou and Xianglong Liu and Bryan Dai},
  title = {{ProductWebBench: Evaluating Website Continuity Beyond Task Completion}},
  year = {2026},
  eprint = {2610.31118},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.31118},
}
Figure