ViBench 2.0: A Benchmark for Long-Horizon Software Building

Daniel Furman, Preeya Kirani, Ibrahim Sheikh, Zhen Li, Peter Zhong, Michele CatastaarXiv 2026 · arXiv

TL;DR

ViBench 2.0 is a new benchmark from Replit designed to evaluate AI agents on "long-horizon" software engineering—building complete, full-stack web applications from scratch over dozens of successive feature requests. The benchmark reveals a significant "reliability gap": while top models can pass nearly 98% of individual test plans, they only deliver a fully functional, 100% working product in at most 65% of builds.

What Problem Does This Paper Address?

Traditional coding benchmarks like HumanEval or MBPP focus on isolated, single-function tasks that are increasingly "saturated" (models are hitting 90%+ accuracy). Even newer agentic benchmarks often only ask an agent to fix a single bug or add one feature to an existing codebase.

However, real-world "vibe coding"—a term coined by Andrej Karpathy to describe building software through natural language iteration—involves growing a product over a long conversation. A small error in an early feature can compound, leading to a broken product by the final request. ViBench 2.0 addresses this by requiring agents to build 17 different full-stack products (clones of apps like Uber, Discord, and Jira) from an empty project, judging them only after the final feature is implemented.

Methods

The ViBench 2.0 pipeline (see Figure 1) follows a "greenfield" build process:

  1. Building: An agent receives a series of Product Requirement Documents (PRDs) one at a time in a single conversation. It must manage its own context, summarize long histories, and maintain a working PostgreSQL database and application scripts.
  2. Seeding: Once the build is "finished," a separate seeding agent inspects the app and populates it with necessary data (e.g., creating user accounts or ride histories) to prepare for testing.
  3. Grading: A browser-based agent (using Playwright) interacts with the finished product's UI just like a human user would. It follows specific test plans to verify outcomes (e.g., "Can I split a fare?") rather than checking specific code implementations.
  4. Reliability Protocol: To ensure the "all-or-nothing" score is fair, the system re-grades failed plans up to five times until a majority consensus is reached, minimizing "wrong fails" caused by flaky browser interactions.
Figure 1: The ViBench 2.0 pipeline. The builder receives one PRD per stage and never sees the test plans; a seeding agent then loads each plan’s data, and a browser agent grades the finished app.

Datasets

The benchmark consists of 17 full-stack web products modeled on widely used software.

  • Public Set (8 products): Clones of Amazon, Asana, Discord, Figma, GitHub, Google Docs, Jira, and Uber. These are released with PRDs and test plans for the community.
  • Held-out Set (9 products): Kept private to prevent models from training on the test data, ensuring "clean" evaluations.
  • Scale: Across all products, there are 241 feature requests and 544 test plans. Features are ordered by a dependency graph (see Figure 2 for the Uber example) to ensure logical progression.
Figure 2: Feature dependencies of the product modeled on Uber, numbered in build order; an arrow from A to B means that B builds on A. Orange marks features two plans check together: earnings must show the car and cancellation fees, and a split fare must not alter them.

Key Insights and Ideas

  • The Reliability Gap: The most capable models are "almost there" but lack consistency. GPT-6.1 Sol passes ~97% of individual test plans but only achieves a 64.7% pass@1 (the rate of builds where every plan passes).
  • Leading vs. Lagging Metrics: The authors found that "plan pass rate" (percentage of tests passed) is a leading indicator. As models improve, this rate rises steadily before the pass@1 score (the "perfect build" rate) starts to move at all.
  • Context is King: In long-horizon tasks (conversations exceeding 200k tokens), how an agent compacts and summarizes its history is a "first-order concern." Poor summarization leads to agents "forgetting" how to build upon earlier features.
  • Visual Quality vs. Functionality: Interestingly, models like Claude 3.5 Sonnet produced functional code but "plain" UIs, while GPT-6 and Sol produced "polished, product-like pages" under the same minimal prompts.

Key Results

The benchmark clearly separates frontier models from the rest of the pack (see Figure 4).

  • Frontier Performance: GPT-6.1 Sol leads with a 64.7% pass@1, followed by GPT-6 Astra at 60.3%. Claude 3.5 Opus follows at 48.5%.
  • Open-Weight Gap: Currently, no open-weight model (including DeepSeek V4.1 Flash and GLM 5.3) managed to deliver a 100% working product (0.0% pass@1), though DeepSeek showed promise with a 77.4% plan pass rate.
  • Rapid Improvement: The benchmark tracks the fast pace of AI development. Across three releases of the Sol family, pass@1 on public products jumped from 3% to 53% while build costs were cut by more than half (see Figure 3).
Figure 4: Main model comparison. The three strongest models pass 94 to 98% of test plans but deliver a fully working product in only 49 to 65% of builds.
Figure 3: Performance across GPT versions. Newer releases pass more test plans, but pass@1 only rises once the plan pass rate exceeds ~70%.

Limitations and Future Work

  • Contamination: While 9 products are held out, the 8 public products could eventually leak into training sets. The authors use "canary strings" to help researchers filter this data.
  • Cost: Running a full evaluation is expensive, costing roughly $85 per build (or ~$1,440 for a full 17-product round). The team is working on a "lighter" variant to lower this barrier.
  • Scope: The benchmark currently only covers full-stack web apps in English with PostgreSQL backends. It does not yet evaluate visual design quality, application speed, or the cost of the resulting software.
  • Human-in-the-loop: Unlike real use, the agent receives no bug reports or feedback between requests; it must get every step right the first time to succeed.

Summary updated

Abstract

On Replit, our agentic engineering platform, users now build entire products without writing code, through successive feature requests to an agent. Yet existing coding benchmarks evaluate single, fully specified tasks, and many are near saturation. We introduce ViBench 2.0, a benchmark for long-horizon software building that we use to select our platform's models. Its 17 full-stack web products, modeled on widely used software, grow over 241 feature requests and are judged only when finished, as a user would judge them: a browser agent runs 544 test plans, and a build passes only if every plan does, under a grading protocol validated to keep this strict score reliable. Evaluating nine models, we find a reliability gap: the three strongest pass up to 98% of test plans yet deliver a fully working product in at most 65% of builds, and no open-weight model delivers one. The gap is closing fast: across three releases of one model family, pass@1 on the public products rises from 3% to 53% while the cost per build falls by more than half. We release eight products with the harness and grader, hold out nine, and report lessons from production use.
BibTeX
@misc{furman2026vibench,
  author = {Daniel Furman and Preeya Kirani and Ibrahim Sheikh and Zhen Li and Peter Zhong and Michele Catasta},
  title = {{ViBench 2.0: A Benchmark for Long-Horizon Software Building}},
  year = {2026},
  eprint = {2611.05620},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05620},
}
Figure