TL;DR
RealBench is a new repository-level code generation benchmark designed to mimic real-world software development by providing LLMs with both natural language requirements and structured UML diagrams. Evaluating six state-of-the-art LLMs reveals a stark reality: while models are excellent at scaffolding the requested architecture, their functional accuracy is low, with the best model achieving an average test pass rate of just 19.39%.
What Problem Does This Paper Address?
Most mainstream code generation benchmarks (like HumanEval) evaluate Large Language Models (LLMs) on short, isolated tasks—typically writing a single function based on a text prompt. Even newer repository-level benchmarks usually only provide raw text requirements.
In the software industry, however, developers rarely code straight from a vague text description. Instead, they work with structured system designs, architectural specifications, and Unified Modeling Language (UML) diagrams. Because existing benchmarks ignore this structural context, their scores fail to reflect how well LLMs can actually automate real-world enterprise software development. The authors introduce RealBench to bridge this gap.
Methods
To systematically evaluate LLMs under realistic conditions, the authors designed three distinct repository-level generation strategies:
- Holistic Generation: Asking the LLM to generate the entire codebase all at once based on the requirements and UML design.
- Incremental Generation: Prompting the LLM to generate the repository in a file-by-file manner to minimize context length and long-range coherence issues.
- Retrieval-Augmented Generation (RAG): Generating file-by-file, but actively feeding previously generated, relevant files back into the prompt based on a directed acyclic graph of file import relationships.
The evaluation framework operates at two granularities:
- Repository-level: Evaluates how well the codebase matches the original intent using an automated repository-understanding agent (DeepWiki) paired with human expert verification.
- Class-level: Measures code correctness using three metrics: Completion rate (did it generate the modules?), Execution pass rate (does it compile/run without syntax errors?), and Test pass rate (does it pass the functional test suite?).
Datasets
The authors constructed the RealBench dataset consisting of:
- 61 real-world Python repositories spanning 20 diverse programming domains (e.g., Internet, Security, Databases).
- Data Contamination Mitigation: All repositories were sourced from GitHub after December 2024, deliberately placing them past the training cut-off dates of the evaluated LLMs.
- Four Difficulty Levels: Categorized by lines of code (LOC), ranging from Level-1 (0–500 LOC) to Level-4 (≥2000 LOC).
- A Comprehensive Test Suite: Features an average of 50 test cases per repository, achieving a high average line coverage of 79.76%.
Key Insights and Ideas
- Great Architects, Poor Builders: LLMs achieved high completion rates (often >50%), meaning they are highly capable of reading UML diagrams and creating the correct classes, methods, and file structures. However, their execution and test pass rates were drastically lower due to severe internal logic and grammar errors.
- More Context Isn't Always Better (The RAG Penalty): Counterintuitively, the RAG strategy performed worse than pure incremental generation. The authors found that feeding previously generated code back into the prompt overwhelmed the LLMs' context windows and frequently caused them to incorrectly blend separate modules together.
- Size Dictates Strategy: For smaller codebases (<1000 LOC), generating the entire repository at once (Holistic) yields the best results. For larger, complex repositories, breaking the task down file-by-file (Incremental) is significantly more effective.
- The Crucial Role of Low-Level Design: When the authors removed detailed class diagrams and only provided high-level package diagrams, LLM performance plummeted to near zero. High-level architecture is not enough; LLMs strictly depend on explicit property and method definitions to generate runnable code.
Key Results
- Low Overall Performance: The absolute best average test pass rate across all models and strategies was a meager 19.39%, highlighting that repo-level generation remains an open challenge.
- The Scalability Wall: Performance drops sharply as software complexity grows. On average, models achieved over a 40% test pass rate on Level-1 repositories (<500 LOC), but this plummeted to under 15% for Level-4 repositories (>2000 LOC).
- Model Standouts: Among open-source models, DeepSeek-V3 performed the best. In the proprietary space, GPT-4o and Claude-Sonnet-4 significantly outperformed Gemini-2.5-Flash.
- Dominant Errors: An analysis of failed tests showed that
AssertionError(54.3%) andTypeError(14.3%) were the top two culprits, accounting for over 68% of all generation failures.
Limitations and Future Work
Currently, RealBench is limited exclusively to Python repositories. Additionally, because the benchmark relies on prompting templates, the results are inherently tied to how well the model follows specific formatting instructions.
The authors plan to expand RealBench to support other major programming languages (like Java and Go) and intend to explore more advanced prompting and decoding strategies to help LLMs better navigate complex software architectures.
Summary updated