Traditional code generation benchmarks often focus on isolated, self-contained functions described by clear docstrings. However, as Jimenez et al. argue, professional software engineering requires navigating massive codebases where the necessary context for a task is distributed across thousands of files. When Large Language Models (LLMs) are restricted to a single-file view, their performance drops significantly, often leading to hallucinations of API calls or class instantiations that rely on external definitions Ding et al..
The core challenge of repository-level code modeling is bridging the gap between local code perception and the global architectural logic of a software project.
The Dependency Resolution Problem
In a typical repository, a single function may rely on utility functions, global variables, or class hierarchies defined elsewhere. Karmakar et al. and Robbes found that a significant majority of method calls in large-scale Java projects occur outside the immediate class, creating a performance ceiling for models that only analyze local context.
Without access to these distributed dependencies, models suffer from "code smells," such as reimplementing existing functions instead of calling them Le Hai et al.. Furthermore, Li et al. demonstrate that providing local file context alone can more than double the success rate of repository-level code generation, highlighting that the primary hurdle is not just logic, but environmental awareness.
Retrieval and Iterative Generation
To address the limitations of fixed context windows, researchers have turned to Retrieval-Augmented Generation (RAG). Zhang et al. proposed RepoCoder, which uses an iterative "retrieval-generation" pipeline. The system uses the model's own initial code predictions to refine its search for relevant snippets across the codebase, effectively using the model's "best guess" to find the actual dependencies it needs.
Similarly, Liu et al. introduced RepoBench, which decomposes the problem into three distinct stages:
- Retrieval: Identifying the right context from disparate files.
- Completion: Using that context to generate code.
- Pipeline Management: Coordinating these tasks end-to-end.
Graph-Based and Structural Context
While text-based retrieval is powerful, code is inherently structured. Ouyang et al. introduced RepoGraph, which builds a line-level dependency graph of an entire repository. By providing models with "ego-graphs"—subgraphs centered on the relevant code—they allow the LLM to navigate complex webs of interdependent modules that simple text sequences cannot capture.
This structural approach is echoed by Karmakar et al., who advocate for the use of call graphs and class hierarchies to resolve identifiers across package boundaries. By treating the repository as a logical, executable entity rather than just a collection of text files, these methods help models maintain consistency with project-wide architectural patterns Jiang et al..
The Limits of Long Context
One might assume that simply expanding the context window of LLMs would solve the repository-level problem. However, Li et al. identified a "32K token wall" in their LongCodeU benchmark. Even models claiming context windows of 128K tokens or more show a sharp decline in reasoning ability once the input exceeds 32,000 tokens.
The authors found that models struggle more with "inter-unit relations"—tracking how data and logic flow between different functions—than with identifying individual components. This suggests that the primary challenge for repository-level understanding is the synthesis of global structures rather than the mere perception of long sequences. As Strich et al. note, capturing this architectural logic is essential for high-level tasks like project-wide documentation or complex feature implementation.