TL;DR
RepoBench is a new, comprehensive benchmark designed to evaluate how well Large Language Models (LLMs) handle repository-level code auto-completion. By introducing tasks for retrieval, completion, and end-to-end pipelines across Python and Java, it bridges the gap between simple single-file benchmarks and the complex, multi-file reality of modern software engineering.
What Problem Does This Paper Address?
Most current AI code assistants are evaluated on their ability to predict the next line of code within a single, isolated file. However, real-world development happens in repositories where logic is spread across dozens or hundreds of files. To provide a helpful suggestion, an AI needs to know how to find relevant "snippets" from other files (cross-file context) and integrate them into its prediction.
The authors identify an "assessment gap": we have plenty of benchmarks for "fill-in-the-blank" coding, but very few that test the entire workflow of a repository-level system—specifically, the ability to retrieve the right context and then generate code based on that context.
Methods
The researchers proposed RepoBench, which breaks down the auto-completion challenge into three interconnected sub-tasks:
- RepoBench-R (Retrieval): Tests the system's ability to search the repository and find the most relevant code snippet needed to complete the current line.
- RepoBench-C (Code Completion): Given the correct context (both in-file and cross-file), can the model predict the next line of code?
- RepoBench-P (Pipeline): An end-to-end test where the system must first retrieve snippets and then use them to generate the completion.
They also defined three difficulty settings:
- Cross-File-First (XF-F): The most difficult; the model must predict a line using a module it hasn't seen yet in the current file.
- Cross-File-Random (XF-R): The model predicts a line involving a module that has appeared before in the file.
- In-File (IF): A baseline check to ensure cross-file context doesn't "break" standard in-file completion.
Datasets
To ensure the benchmark is rigorous and avoids "memorization" (where the model just remembers code it saw during training), the authors used two distinct sources:
- Training Data: Sourced from the
github-codedataset (data cutoff March 2022). - Evaluation (Test) Data: A newly crawled set of GitHub repositories created between February and August 2023. This ensures that even the most recent models haven't seen this code during their pre-training phase.
- Languages: The benchmark covers both Python (simpler syntax) and Java (more verbose, class-heavy).
Key Insights and Ideas
- Retrieval is the Bottleneck: The study found that UniXcoder significantly outperformed lexical methods (like Jaccard similarity) and CodeBERT in finding relevant snippets.
- Context is King (Even if it's messy): One of the most surprising findings was that including any cross-file context—even randomly selected snippets—improved model performance compared to using no cross-file context at all.
- The "Middle" Problem: While testing StarCoder, the authors noticed a "performance dip" with medium-length contexts that wasn't present in OpenAI's Codex. This suggests that some models struggle specifically with certain prompt lengths.
- Snippet Placement Matters: In the end-to-end pipeline, the order of retrieved snippets affects the result. Placing the most relevant snippets closer to the end of the prompt (near the completion point) generally yields better results.
Key Results
- Codex Leads the Pack: OpenAI’s Codex (code-davinci-002) remains the top performer, especially in Java, where it showed a marked lead over StarCoder and CodeGen.
- Python vs. Java: Models generally performed better on Python tasks than Java. The authors attribute this to Python’s simpler syntax and the way function arguments are often defined close to their calls.
- Retrieval Accuracy: On the "Hard" Python retrieval subset, UniXcoder achieved an Accuracy@1 of 17.70%, while a random baseline was only 6.44%.
- Completion Performance: In the 2k context subset for Python, Codex, StarCoder, and CodeGen-16B performed almost identically (around 31-33% Exact Match), but Codex pulled away significantly as the context length increased to 8k.
Limitations and Future Work
The authors acknowledge several areas for growth:
- Context Window Limits: Even "long-context" models have limits. RepoBench-P shows that as repositories grow, the "needle in a haystack" problem of retrieval becomes much harder.
- Next-Line vs. Multi-Line: Currently, the benchmark focuses on predicting the next line. Future versions could expand to multi-line blocks or entire function bodies.
- Model Generalizability: The "performance dip" seen in StarCoder suggests that we still don't fully understand how prompt length and data distribution interact in these models.
RepoBench provides a much-needed "reality check" for AI coding tools, moving the goalposts from simple syntax completion to true repository-level understanding.
Summary updated