TL;DR
Large Language Models (LLMs) for code, like GitHub Copilot, often miss crucial context within a project's repository, leading to inaccurate code completions. RepoFusion introduces a novel training framework that explicitly teaches these models to leverage diverse repository contexts, significantly improving their performance and even rivaling much larger models.
What Problem Does This Paper Address?
Modern code LLMs, despite their impressive capabilities, struggle with understanding the broader context of a software project. This limitation becomes particularly apparent when dealing with proprietary code, work-in-progress projects, or codebases not seen during training. The issue stems from the models' inability to effectively utilize information scattered across a repository, such as imports, parent classes, or files with similar naming conventions. This leads to suboptimal code suggestions and completions, hindering their real-world utility.
Methods
The core innovation of RepoFusion is its training framework designed to integrate multiple relevant repository contexts. It builds upon the Fusion-in-Decoder (FiD) architecture, originally used for question answering, and adapts it for code completion. RepoFusion takes a target code snippet (the "target hole") and its surrounding code, then augments this with various "repo contexts" retrieved from the repository. Each repo context, along with its surrounding context, is encoded separately. The decoder then jointly attends to these encoded representations to produce a more accurate code completion.
The paper explores different strategies for selecting and ordering these repo contexts, including:
- T-Rank: Truncating ranked prompt proposals to a fixed length.
- T-Rand: Randomly ordering truncated prompt proposals.
- NT-Rank: Using entire prompt proposals without truncation, generating multiple contexts if needed.
- NT-Prior-Last: Similar to NT-Rank, but always placing the "prior" context (context from the current file before the target hole) last.
Datasets
The authors introduce Stack-Repo, a new dataset comprising 200 near-deduplicated Java repositories with permissive licenses. This dataset is augmented with three types of repository contexts for each target hole. The dataset is split into training, validation, and testing sets.
Key Insights and Ideas
- Repository context is crucial: Explicitly training models to incorporate repository-level information dramatically improves code completion accuracy, even for smaller models.
- Diversity of contexts matters: Leveraging information from multiple, diverse sources within a repository is key to RepoFusion's effectiveness. Simply repeating contexts from the same source yields diminishing returns.
- Ordering of contexts can be important: Strategies like NT-Prior-Last, which place the current file's context last, show superior performance, suggesting that the model benefits from attending to the most immediate context last.
- Longer contexts are generally better, but with diminishing returns: Increasing the length of individual repo contexts improves performance up to a point, after which further increases yield little benefit.
- Finetuning the base model is essential: Pre-trained models like CodeT5 perform poorly on code completion tasks without finetuning on code-specific objectives. Finetuning significantly boosts their performance, and RepoFusion further enhances this.
Key Results
RepoFusion, a 220M parameter model, significantly outperforms larger models like CodeGen-16B-multi (73x larger) on single-line code completion. It also closely matches the performance of the much larger StarCoderBase (70x larger), which was trained with a Fill-in-the-Middle objective. Specifically, RepoFusion achieves a success rate of 77.32% with N=32 and l=768, while StarCoderBase achieves 79.79% with post+prior context. Ablation studies demonstrate the impact of various design choices, highlighting the effectiveness of the NT-Prior-Last strategy and the importance of appending surrounding context.
Limitations and Future Work
The primary limitation acknowledged is computational scalability: RepoFusion's inference time scales linearly with the number of repository contexts (N), making it slower for larger N. The authors suggest leveraging optimization techniques like FiDO as a potential solution. They also note that generated code can sometimes be difficult to understand or debug, and excessive reliance might lead to users overlooking errors or becoming overly confident.
Future work could explore extending RepoFusion to other code-related tasks like bug repair, pull request merging, and software documentation generation. Further research into more efficient context integration and handling of complex repository structures is also suggested.
Summary updated