TL;DR
Large language models (LLMs) powering code assistants often struggle when working in private or unfamiliar repositories because they lack global project context. RepoFusion demonstrates that training a compact 220M-parameter model with a Fusion-in-Decoder architecture to ingest multiple repository-level contexts allows it to dramatically outperform models over 70× larger (like CodeGen-16B) and closely rival state-of-the-art 15.5B-parameter giants like StarCoderBase.
What Problem Does This Paper Address?
When developers write code, they rarely work in isolation within a single file. Useful completions frequently depend on class definitions, utility functions, or API usage defined across other files in the repository. While existing LLMs (like GitHub Copilot) excel at general programming, they stumble on proprietary or in-progress repositories containing unseen internal APIs and project-specific conventions.
Prior approaches tried to address this during inference by retrieving snippets from the repository and stuffing them into the prompt. However, models without explicit training struggle to integrate multiple disparate contexts scattered throughout a codebase.
Methods
RepoFusion adapts the Fusion-in-Decoder (FiD) framework—originally developed for open-domain question answering—to single-line code completion:
- Context Retrieval: To fetch project context, the authors extract $N$ repository contexts (RCs) using structured Prompt Proposals (PP) (heuristics targeting imports, parent classes, method declarations, and similar filenames), comparing them against standard BM25 and nearest-neighbor dense retrieval (RandomNN).
- Encoding & Fusion: Each repository context is concatenated with the local surrounding context (two lines above and below the cursor) and encoded independently using a Transformer encoder.
- Joint Decoding: The decoder cross-attends across all concatenated encoded representations to generate the missing code token-by-token.
- Context Ordering: The authors introduce NT-Prior-Last (Not Truncated, Prior Context Last), where top-ranked contexts are kept intact rather than truncated, and the in-file code preceding the cursor (the prior) is fed last so the decoder can attend to it most directly.
The base model is a 220M-parameter CodeT5-base, first finetuned on next-token prediction across Java repositories before RepoFusion training.
[ Repo Context 1 + Local Context ] ──> Encoder ──┐
[ Repo Context 2 + Local Context ] ──> Encoder ──┼─> [ Concat ] ──> Decoder ──> Prediction
[ Prior File Context + Local ] ──> Encoder ──┘
Datasets
The authors introduce Stack-Repo, derived from the Java subset of The Stack V1.1 (permissively licensed, near-deduplicated):
- Size: 200 Java repositories split into Train (100 repos, 435,890 target holes), Validation (50 repos, 220,615 holes), and Test (50 repos, 159,822 holes).
- Target Holes: Single-line completion targets created by cutting a line at a randomly selected delimiter token.
- Enrichment: Pre-computed with three retrieval contexts (Prompt Proposals, BM25, and RandomNN).
Key Insights and Ideas
- Architecture over parameter scale: Rather than scaling model parameters to hundreds of billions, giving a small model a structured way to read and attend to project-level context yields comparable or superior performance at a fraction of the compute.
- Domain heuristics beat standard search: Deterministic, syntax-aware Prompt Proposals (capturing imports, inheritance, and API signatures) clearly outperformed BM25 and CodeBERT dense embeddings for repository retrieval.
- Complete context > many truncated snippets: Feeding the full text of high-relevance files (
NT-Rank) outperformed stuffing the context window with fragmented, truncated snippets from many files (T-Rank). - Decoder ordering matters: Positioning the file's immediate prior context as the final block (
NT-Prior-Last) delivered the best results, as the decoder seamlessly bridges the immediate prefix into the completion.
Key Results
Evaluation used exact string match (Success Rate) on the target hole:
- Outperforming 70× larger models: RepoFusion ($N=32, l=768$; 220M parameters) achieved a 77.32% success rate on the test set. In comparison, CodeGen-16B reached only 54.09% (with prior+post context), and CodeT5-large reached 52.43%.
- Fixed context efficiency: Even when restricted to a 2,048-token context window ($N=4, l=512$), RepoFusion scored 65.96%, beating all 2,048-token baselines by at least 9%.
- Matching StarCoderBase: On a 12,500-example subset, RepoFusion ($N=32, l=2500$) achieved 78.33%, coming within striking distance of the 15.5B parameter StarCoderBase (79.79%), despite StarCoderBase being trained on ~1 trillion tokens with Fill-in-the-Middle (FiM).
Limitations and Future Work
- Scope restricted to single-line completions: The study does not evaluate multi-line completions, full function synthesis, or multi-turn conversational edits.
- Language specificity: Experiments and heuristic prompt proposals were built and evaluated strictly on Java repositories.
- Downstream software engineering tasks: The authors highlight extending RepoFusion to larger-scale code operations, such as automated bug fixing, pull request resolution, and documentation generation, as exciting future directions.
Summary updated