RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices

Jia Li, Hongyi Deng, Yiran Zhang, Kechi Zhang, Tianqi Shao, Tiankuo Zhao, Weinan Wang, Zhi Jin, Ge Li, Yang Liu, Yingtao Fang, Yihong Dong

TL;DR

RealBench is a new repository-level code generation benchmark designed to mimic real-world software development by providing LLMs with both natural language requirements and structured UML diagrams. Evaluating six state-of-the-art LLMs reveals a stark reality: while models are excellent at scaffolding the requested architecture, their functional accuracy is low, with the best model achieving an average test pass rate of just 19.39%.

What Problem Does This Paper Address?

Most mainstream code generation benchmarks (like HumanEval) evaluate Large Language Models (LLMs) on short, isolated tasks—typically writing a single function based on a text prompt. Even newer repository-level benchmarks usually only provide raw text requirements.

In the software industry, however, developers rarely code straight from a vague text description. Instead, they work with structured system designs, architectural specifications, and Unified Modeling Language (UML) diagrams. Because existing benchmarks ignore this structural context, their scores fail to reflect how well LLMs can actually automate real-world enterprise software development. The authors introduce RealBench to bridge this gap.

An example of a RealBench task containing requirements and UML diagrams

Methods

To systematically evaluate LLMs under realistic conditions, the authors designed three distinct repository-level generation strategies:

  1. Holistic Generation: Asking the LLM to generate the entire codebase all at once based on the requirements and UML design.
  2. Incremental Generation: Prompting the LLM to generate the repository in a file-by-file manner to minimize context length and long-range coherence issues.
  3. Retrieval-Augmented Generation (RAG): Generating file-by-file, but actively feeding previously generated, relevant files back into the prompt based on a directed acyclic graph of file import relationships.

The evaluation framework operates at two granularities:

  • Repository-level: Evaluates how well the codebase matches the original intent using an automated repository-understanding agent (DeepWiki) paired with human expert verification.
  • Class-level: Measures code correctness using three metrics: Completion rate (did it generate the modules?), Execution pass rate (does it compile/run without syntax errors?), and Test pass rate (does it pass the functional test suite?).

Datasets

The authors constructed the RealBench dataset consisting of:

  • 61 real-world Python repositories spanning 20 diverse programming domains (e.g., Internet, Security, Databases).
  • Data Contamination Mitigation: All repositories were sourced from GitHub after December 2024, deliberately placing them past the training cut-off dates of the evaluated LLMs.
  • Four Difficulty Levels: Categorized by lines of code (LOC), ranging from Level-1 (0–500 LOC) to Level-4 (≥2000 LOC).
  • A Comprehensive Test Suite: Features an average of 50 test cases per repository, achieving a high average line coverage of 79.76%.

Key Insights and Ideas

  • Great Architects, Poor Builders: LLMs achieved high completion rates (often >50%), meaning they are highly capable of reading UML diagrams and creating the correct classes, methods, and file structures. However, their execution and test pass rates were drastically lower due to severe internal logic and grammar errors.
  • More Context Isn't Always Better (The RAG Penalty): Counterintuitively, the RAG strategy performed worse than pure incremental generation. The authors found that feeding previously generated code back into the prompt overwhelmed the LLMs' context windows and frequently caused them to incorrectly blend separate modules together.
  • Size Dictates Strategy: For smaller codebases (<1000 LOC), generating the entire repository at once (Holistic) yields the best results. For larger, complex repositories, breaking the task down file-by-file (Incremental) is significantly more effective.
  • The Crucial Role of Low-Level Design: When the authors removed detailed class diagrams and only provided high-level package diagrams, LLM performance plummeted to near zero. High-level architecture is not enough; LLMs strictly depend on explicit property and method definitions to generate runnable code.

Key Results

  • Low Overall Performance: The absolute best average test pass rate across all models and strategies was a meager 19.39%, highlighting that repo-level generation remains an open challenge.
  • The Scalability Wall: Performance drops sharply as software complexity grows. On average, models achieved over a 40% test pass rate on Level-1 repositories (<500 LOC), but this plummeted to under 15% for Level-4 repositories (>2000 LOC).
  • Model Standouts: Among open-source models, DeepSeek-V3 performed the best. In the proprietary space, GPT-4o and Claude-Sonnet-4 significantly outperformed Gemini-2.5-Flash.
  • Dominant Errors: An analysis of failed tests showed that AssertionError (54.3%) and TypeError (14.3%) were the top two culprits, accounting for over 68% of all generation failures.

Limitations and Future Work

Currently, RealBench is limited exclusively to Python repositories. Additionally, because the benchmark relies on prompting templates, the results are inherently tied to how well the model follows specific formatting instructions.

The authors plan to expand RealBench to support other major programming languages (like Java and Go) and intend to explore more advanced prompting and decoding strategies to help LLMs better navigate complex software architectures.

Summary updated

Abstract

Writing code requires significant time and effort in software development. To automate this process, researchers have made substantial progress using Large Language Models (LLMs) for code generation. Many benchmarks like HumanEval and EvoCodeBench have been created to evaluate LLMs by requiring them to generate code from natural language requirements. However, in enterprise applications and team development, developers typically write code based on structured designs or specifications rather than raw natural language descriptions. This gap between existing benchmarks and real industry development practices means that current benchmark scores may not accurately reflect how much code generation can help automate software development tasks. To address this gap, we propose RealBench, a repository-level code generation benchmark aligned with real-world industry software development practices. Each example includes both natural language requirements and UML diagrams as system design, matching how developers typically receive specifications. Based on the constructed benchmarks, we conduct a systematic evaluation of advanced LLMs' code generation capabilities when provided with structured system designs. The experimental results reveal key insights in current LLMs' capabilities for repo-level code generation aligned with real-world software development practices. First, we notice that regarding repo-level code generation, LLMs show much worse performance and there are significant performance gaps among LLMs. Second, LLMs are good at finding and creating modules defined in UML diagrams, but the quality of generated modules is often poor due to grammar and logic errors. Third, generating the entire repository at once is the best generation strategy on smaller repositories, while generating a complex repository with the module-by-module strategy works better compared to other strategies.
BibTeX
@article{li2026realbench,
  author = {Jia Li and Hongyi Deng and Yiran Zhang and Kechi Zhang and Tianqi Shao and Tiankuo Zhao and Weinan Wang and Zhi Jin and Ge Li and Yang Liu and Yingtao Fang and Yihong Dong},
  title = {{RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices}},
  journal = {Proc. ACM Softw. Eng.},
  year = {2026},
  doi = {10.1145/3797106},
}

Loading related content...

Figure