TL;DR
Large Language Models (LLMs) are increasingly used for code generation, but existing benchmarks don't reflect real-world software development. This paper introduces DevEval, a new benchmark that better captures the complexities of practical projects, including realistic program distributions, sufficient dependencies, and substantial project contexts. Their evaluation reveals that current LLMs, while improving with context, still struggle with the intricacies of real-world code generation.
What Problem Does This Paper Address?
The core problem is the disconnect between existing benchmarks for evaluating code-generating LLMs and the reality of practical software development. Many current benchmarks, like HumanEval, focus on "standalone" functions that don't rely on other parts of a project. However, in real-world projects, a staggering 73.8% of functions are "non-standalone," meaning they depend on other code within the project. This gap leads to an overestimation of LLM capabilities, as they may perform well on isolated tasks but falter when faced with the interconnectedness and scale of actual software. The authors highlight three key gaps: unrealistic program distributions (too many standalone functions), insufficient dependencies (not enough inter-code relationships), and small-scale project contexts (models aren't exposed to enough surrounding code).
Methods
The paper introduces DevEval, a new benchmark designed to bridge the gap between academic evaluation and practical software development. The creation of DevEval involved a rigorous pipeline:
- Project Selection: Crawling the top 10 programming topics on PyPI and selecting the top 50 projects by stars from each topic, resulting in 500 initial projects.
- Function Parsing: Extracting functions, their signatures, reference code, and surrounding project contexts from these projects.
- Test Construction: Leveraging existing test cases from the projects and organizing them using the Pytest framework.
- Human Annotation: Engaging 13 annotators to manually write natural language requirements and identify reference dependencies for the extracted functions. This step was crucial for ensuring clarity and accuracy, as original code comments are often insufficient.
- Benchmark Construction: Finally, DevEval was constructed by carefully selecting samples to match the real-world distribution of standalone (26.2%) and non-standalone (73.8%) functions observed in the 500 practical projects.
DevEval includes six components per sample: Function Signature, Requirement, Project Contexts, Reference Code, Reference Dependency, and Test Cases.
Datasets
- Existing Benchmarks: HumanEval, MBPP, Concode, CoderEval, etc. (used for comparison).
- DevEval: A newly introduced benchmark comprising 2,690 samples from 119 practical projects across 10 domains. It aims to represent real-world program distributions, sufficient dependencies (intra-class, intra-file, cross-file), and large-scale project contexts.
Key Insights and Ideas
- Real-world code is interconnected: The 73.8% non-standalone function rate in practical projects is a critical insight, highlighting the limitations of benchmarks that focus on isolated functions.
- Context is king (but long context is hard): LLMs significantly improve their code generation performance when provided with project contexts. However, even with extensive contexts, their performance is still far from perfect, suggesting they struggle to effectively process and utilize long, heterogeneous codebases.
- Dependencies matter: The type and number of dependencies are crucial. DevEval includes a richer set of dependencies (intra-class, intra-file, cross-file) than previous benchmarks, and LLMs show varying abilities in generating them.
- Human-written requirements are vital: Original code comments are often too vague for effective code generation. Manually crafted requirements, which are detailed and capture the code's purpose, significantly aid LLMs.
- Current LLMs hallucinate: Even with context, LLMs can generate non-existent functions, indicating a struggle with understanding and integrating information from complex project structures.
Key Results
- Performance Gap: LLMs exhibit significantly lower Pass@1 scores on DevEval compared to popular benchmarks like HumanEval. For instance, gpt-3.5-turbo-1106 achieved a Pass@1 of 73 on HumanEval but only 42 on DevEval.
- Contextual Improvement: Providing project contexts dramatically improves LLM performance. For gpt-3.5-turbo-1106, Pass@1 increased from 18.03 (baseline) to 44.83% with "Local + Oracle" contexts.
- Standalone vs. Non-standalone: Project contexts are particularly crucial for generating non-standalone functions, showing a substantial improvement in Pass@1 when contexts are introduced.
- Dependency Recall: LLMs show improved Recall@k with contexts, but still struggle with generating cross-file dependencies effectively.
- Context Length Limitation: Even with advanced context compression techniques, LLMs' performance plateaus, suggesting current context windows and processing capabilities are insufficient for the length of real-world projects.
Limitations and Future Work
- Context Length: The primary limitation identified is the sheer length of project contexts, which often exceeds the context window of even the most advanced LLMs. This leads to LLMs ignoring relevant information or hallucinating.
- Context Heterogeneity: LLMs are trained on continuous text, but project contexts are often composed of discrete code snippets from various files, creating a mismatch that hinders understanding.
- Data Leakage: While the authors believe the risk is negligible due to manual requirement writing and evaluation on unseen projects, they acknowledge the potential for training data overlap.
- Parser Bias: The automated dependency extraction parser may miss some runtime dependencies, leading to a slight bias in Recall@k scores.
- Future Work:
- Context Compression: Developing more adaptive techniques to retrieve only relevant contexts.
- Context Understanding: Exploring new pre-training objectives or prompting strategies to improve LLMs' ability to understand long and heterogeneous contexts.
- Accelerating Generation: Improving the speed of code generation with long contexts.
Summary updated