TL;DR
ViBench 2.0 is a new benchmark from Replit designed to evaluate AI agents on "long-horizon" software engineering—building complete, full-stack web applications from scratch over dozens of successive feature requests. The benchmark reveals a significant "reliability gap": while top models can pass nearly 98% of individual test plans, they only deliver a fully functional, 100% working product in at most 65% of builds.
What Problem Does This Paper Address?
Traditional coding benchmarks like HumanEval or MBPP focus on isolated, single-function tasks that are increasingly "saturated" (models are hitting 90%+ accuracy). Even newer agentic benchmarks often only ask an agent to fix a single bug or add one feature to an existing codebase.
However, real-world "vibe coding"—a term coined by Andrej Karpathy to describe building software through natural language iteration—involves growing a product over a long conversation. A small error in an early feature can compound, leading to a broken product by the final request. ViBench 2.0 addresses this by requiring agents to build 17 different full-stack products (clones of apps like Uber, Discord, and Jira) from an empty project, judging them only after the final feature is implemented.
Methods
The ViBench 2.0 pipeline (see Figure 1) follows a "greenfield" build process:
- Building: An agent receives a series of Product Requirement Documents (PRDs) one at a time in a single conversation. It must manage its own context, summarize long histories, and maintain a working PostgreSQL database and application scripts.
- Seeding: Once the build is "finished," a separate seeding agent inspects the app and populates it with necessary data (e.g., creating user accounts or ride histories) to prepare for testing.
- Grading: A browser-based agent (using Playwright) interacts with the finished product's UI just like a human user would. It follows specific test plans to verify outcomes (e.g., "Can I split a fare?") rather than checking specific code implementations.
- Reliability Protocol: To ensure the "all-or-nothing" score is fair, the system re-grades failed plans up to five times until a majority consensus is reached, minimizing "wrong fails" caused by flaky browser interactions.
Datasets
The benchmark consists of 17 full-stack web products modeled on widely used software.
- Public Set (8 products): Clones of Amazon, Asana, Discord, Figma, GitHub, Google Docs, Jira, and Uber. These are released with PRDs and test plans for the community.
- Held-out Set (9 products): Kept private to prevent models from training on the test data, ensuring "clean" evaluations.
- Scale: Across all products, there are 241 feature requests and 544 test plans. Features are ordered by a dependency graph (see Figure 2 for the Uber example) to ensure logical progression.
Key Insights and Ideas
- The Reliability Gap: The most capable models are "almost there" but lack consistency. GPT-6.1 Sol passes ~97% of individual test plans but only achieves a 64.7%
pass@1(the rate of builds where every plan passes). - Leading vs. Lagging Metrics: The authors found that "plan pass rate" (percentage of tests passed) is a leading indicator. As models improve, this rate rises steadily before the
pass@1score (the "perfect build" rate) starts to move at all. - Context is King: In long-horizon tasks (conversations exceeding 200k tokens), how an agent compacts and summarizes its history is a "first-order concern." Poor summarization leads to agents "forgetting" how to build upon earlier features.
- Visual Quality vs. Functionality: Interestingly, models like Claude 3.5 Sonnet produced functional code but "plain" UIs, while GPT-6 and Sol produced "polished, product-like pages" under the same minimal prompts.
Key Results
The benchmark clearly separates frontier models from the rest of the pack (see Figure 4).
- Frontier Performance: GPT-6.1 Sol leads with a 64.7% pass@1, followed by GPT-6 Astra at 60.3%. Claude 3.5 Opus follows at 48.5%.
- Open-Weight Gap: Currently, no open-weight model (including DeepSeek V4.1 Flash and GLM 5.3) managed to deliver a 100% working product (
0.0% pass@1), though DeepSeek showed promise with a 77.4% plan pass rate. - Rapid Improvement: The benchmark tracks the fast pace of AI development. Across three releases of the Sol family,
pass@1on public products jumped from 3% to 53% while build costs were cut by more than half (see Figure 3).
Limitations and Future Work
- Contamination: While 9 products are held out, the 8 public products could eventually leak into training sets. The authors use "canary strings" to help researchers filter this data.
- Cost: Running a full evaluation is expensive, costing roughly $85 per build (or ~$1,440 for a full 17-product round). The team is working on a "lighter" variant to lower this barrier.
- Scope: The benchmark currently only covers full-stack web apps in English with PostgreSQL backends. It does not yet evaluate visual design quality, application speed, or the cost of the resulting software.
- Human-in-the-loop: Unlike real use, the agent receives no bug reports or feedback between requests; it must get every step right the first time to succeed.
Summary updated