Repository Level Benchmarks for Code Intelligence

153 concepts143 papers3 proposals

Early evaluations of language models on coding tasks relied heavily on isolated, single-function puzzles. Benchmarks like competitive programming problems or standalone syntax tasks tested a model's ability to generate algorithmic logic in a vacuum. However, real-world software engineering rarely happens within a single isolated function. It requires navigating massive codebases, understanding cross-file dependencies, interpreting real issue reports, and maintaining state across millions of tokens.

To bridge this gap, a major research direction has emerged focused on constructing repository-level and context-rich benchmarks. These datasets shift the evaluation paradigm from simple code generation to complex, end-to-end software engineering tasks like multi-file debugging, repository-aware code completion, and automated code review.


The Anatomy of Repository-Level Context

Why do models that excel at standalone puzzles struggle when dropped into a real software project? The answer lies in the distribution of context. In a real repository, a bug or a feature request cannot be resolved without understanding the surrounding environment—such as local utility functions, custom APIs, and project-specific configurations.

To evaluate models under these realistic conditions, researchers have built benchmarks directly from entire code repositories:

  • Long-Context Windows: Stefano Rando et al. introduced LongCodeBench, which tests models on context windows up to one million tokens using real GitHub repositories. This forces models to locate relevant information across hundreds of files rather than processing a cleanly packaged prompt.
  • Repository-Aware Completion: Instead of predicting the next line of code based solely on the current file, benchmarks like Stack-Repo, introduced by Disha Shrivastava et al., provide models with diverse repository contexts (such as sibling files and imported modules) to evaluate how well they leverage project-wide architecture.
  • End-to-End Repair: Benchmarks like RepoFixEval, developed by Tao Sun et al., require models to ingest an entire codebase and an associated issue report, localize the bug across multiple files, and generate a compilable patch.

By embedding the target task within a web of interconnected files, these benchmarks evaluate a model's capacity for structural comprehension rather than just token prediction.


Automated and High-Fidelity Curation Mechanisms

Manually writing thousands of repository-level programming puzzles is prohibitively expensive and risks creating synthetic problems that do not mirror real-world software defects. To scale these datasets, researchers leverage automated curation pipelines that mine existing developer workflows.

1. Mining Pull Requests and Issue Trackers

The most direct way to capture real software engineering is to look at how humans fix bugs. When a developer resolves an issue on GitHub, they leave behind a trace: an issue description, a pull request (PR) containing the code changes, and a set of unit tests that verify the fix.

Aditya Bharat Soni et al. utilized this pattern in SWE-Tester to build a dataset for training models to generate issue reproduction tests. Similarly, Shuai Lu et al. mined hundreds of thousands of pull requests across multiple languages to create CodeReviewer, a dataset aimed at automating code review and comment generation based on contextual code changes.

2. Fuzzing and Crash Log Extraction

For low-level or highly critical systems like the Linux kernel, bugs are often discovered by automated testing tools rather than human issue reports. Alex Mathai et al. demonstrated how to automate benchmark curation for these domains in kGym. By extracting crash reports and reproduction scripts directly from continuous fuzzing logs, the authors created a verifiable ground truth for kernel crash resolution without relying on manual labeling.

3. Programmatic Bug Injection and Static Analysis

Another approach involves taking stable, real-world repositories and programmatically introducing defects. Jingjing Liu et al. used this strategy in RepoDebug, injecting specific categories of bugs into high-quality GitHub repositories to evaluate multi-task debugging. Alternatively, Matthew Jin et al. used static analyzers on historical repository commits to automatically isolate bugs, enriching the dataset with rich metadata like precise bug locations and defect types.


The Necessity of Data Cleaning and Verification

While automated scraping allows for massive scale, raw data from open-source repositories is notoriously noisy. Commits may contain unrelated refactoring, issue descriptions can be misleading, and scraped datasets often contain duplicate or low-quality code.

Lin Shi 0006 et al. demonstrated the severity of this issue by showing that widely used code summarization benchmarks are riddled with noisy data, and that systematic data cleaning drastically alters model evaluation performance.

To combat this, modern repository-level benchmarks incorporate rigorous filtering and human-in-the-loop verification. For example, SWE-bench Verified, highlighted in work by Zhaotian Weng et al., relies on a human-validated subset of software engineering problems to ensure that the evaluation tests are completely stable and that the ground-truth solutions are unambiguous.


Summary

The evolution of code intelligence datasets represents a shift from testing syntax generation to testing software engineering execution. By building benchmarks that encompass entire repositories, leveraging automated pipelines like PR mining and fuzzing logs, and enforcing strict verification standards, this research direction ensures that as language models grow more capable, our benchmarks remain difficult, realistic, and aligned with actual software development practices.

Key Foundational Work

BOAT: an experimental platform for researchers to comparatively and reproducibly evaluate bug localization techniques (2014) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_custom_programming_puzzle_dataset_49befa56,
  title = {custom programming puzzle dataset},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/custom-programming-puzzle-dataset/rdr_49befa56a49741e4b242f944dc194e6c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_custom_programming_puzzle_dataset_49befa56_v1,
  title = {custom programming puzzle dataset},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/custom-programming-puzzle-dataset/rdr_49befa56a49741e4b242f944dc194e6c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake