RepoFusion: Training Code Models to Understand Your Repository

Disha Shrivastava, Denis Kocetkov, Harm de Vries, Dzmitry Bahdanau, Torsten Scholak

TL;DR

Large Language Models (LLMs) for code, like GitHub Copilot, often miss crucial context within a project's repository, leading to inaccurate code completions. RepoFusion introduces a novel training framework that explicitly teaches these models to leverage diverse repository contexts, significantly improving their performance and even rivaling much larger models.

What Problem Does This Paper Address?

Modern code LLMs, despite their impressive capabilities, struggle with understanding the broader context of a software project. This limitation becomes particularly apparent when dealing with proprietary code, work-in-progress projects, or codebases not seen during training. The issue stems from the models' inability to effectively utilize information scattered across a repository, such as imports, parent classes, or files with similar naming conventions. This leads to suboptimal code suggestions and completions, hindering their real-world utility.

Methods

The core innovation of RepoFusion is its training framework designed to integrate multiple relevant repository contexts. It builds upon the Fusion-in-Decoder (FiD) architecture, originally used for question answering, and adapts it for code completion. RepoFusion takes a target code snippet (the "target hole") and its surrounding code, then augments this with various "repo contexts" retrieved from the repository. Each repo context, along with its surrounding context, is encoded separately. The decoder then jointly attends to these encoded representations to produce a more accurate code completion.

The paper explores different strategies for selecting and ordering these repo contexts, including:

  • T-Rank: Truncating ranked prompt proposals to a fixed length.
  • T-Rand: Randomly ordering truncated prompt proposals.
  • NT-Rank: Using entire prompt proposals without truncation, generating multiple contexts if needed.
  • NT-Prior-Last: Similar to NT-Rank, but always placing the "prior" context (context from the current file before the target hole) last.

Datasets

The authors introduce Stack-Repo, a new dataset comprising 200 near-deduplicated Java repositories with permissive licenses. This dataset is augmented with three types of repository contexts for each target hole. The dataset is split into training, validation, and testing sets.

Key Insights and Ideas

  • Repository context is crucial: Explicitly training models to incorporate repository-level information dramatically improves code completion accuracy, even for smaller models.
  • Diversity of contexts matters: Leveraging information from multiple, diverse sources within a repository is key to RepoFusion's effectiveness. Simply repeating contexts from the same source yields diminishing returns.
  • Ordering of contexts can be important: Strategies like NT-Prior-Last, which place the current file's context last, show superior performance, suggesting that the model benefits from attending to the most immediate context last.
  • Longer contexts are generally better, but with diminishing returns: Increasing the length of individual repo contexts improves performance up to a point, after which further increases yield little benefit.
  • Finetuning the base model is essential: Pre-trained models like CodeT5 perform poorly on code completion tasks without finetuning on code-specific objectives. Finetuning significantly boosts their performance, and RepoFusion further enhances this.

Key Results

RepoFusion, a 220M parameter model, significantly outperforms larger models like CodeGen-16B-multi (73x larger) on single-line code completion. It also closely matches the performance of the much larger StarCoderBase (70x larger), which was trained with a Fill-in-the-Middle objective. Specifically, RepoFusion achieves a success rate of 77.32% with N=32 and l=768, while StarCoderBase achieves 79.79% with post+prior context. Ablation studies demonstrate the impact of various design choices, highlighting the effectiveness of the NT-Prior-Last strategy and the importance of appending surrounding context.

Limitations and Future Work

The primary limitation acknowledged is computational scalability: RepoFusion's inference time scales linearly with the number of repository contexts (N), making it slower for larger N. The authors suggest leveraging optimization techniques like FiDO as a potential solution. They also note that generated code can sometimes be difficult to understand or debug, and excessive reliance might lead to users overlooking errors or becoming overly confident.

Future work could explore extending RepoFusion to other code-related tasks like bug repair, pull request merging, and software documentation generation. Further research into more efficient context integration and handling of complex repository structures is also suggested.

Summary updated

Abstract

Despite the huge success of Large Language Models (LLMs) in coding assistants like GitHub Copilot, these models struggle to understand the context present in the repository (e.g., imports, parent classes, files with similar names, etc.), thereby producing inaccurate code completions. This effect is more pronounced when using these assistants for repositories that the model has not seen during training, such as proprietary software or work-in-progress code projects. Recent work has shown the promise of using context from the repository during inference. In this work, we extend this idea and propose RepoFusion, a framework to train models to incorporate relevant repository context. Experiments on single-line code completion show that our models trained with repository context significantly outperform much larger code models as CodeGen-16B-multi ($\sim73\times$ larger) and closely match the performance of the $\sim 70\times$ larger StarCoderBase model that was trained with the Fill-in-the-Middle objective. We find these results to be a novel and compelling demonstration of the gains that training with repository context can bring. We carry out extensive ablation studies to investigate the impact of design choices such as context type, number of contexts, context length, and initialization within our framework. Lastly, we release Stack-Repo, a dataset of 200 Java repositories with permissive licenses and near-deduplicated files that are augmented with three types of repository contexts. Additionally, we are making available the code and trained checkpoints for our work. Our released resources can be found at \url{https://huggingface.co/RepoFusion}.
BibTeX
@misc{shrivastava2023repofusion,
  author = {Disha Shrivastava and Denis Kocetkov and Harm de Vries and Dzmitry Bahdanau and Torsten Scholak},
  title = {{RepoFusion: Training Code Models to Understand Your Repository}},
  year = {2023},
  eprint = {2306.10998},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.10998},
}
Figure