TL;DR
This paper introduces CoRet, a novel retrieval model designed to help developers and AI agents find relevant code snippets within large repositories for editing tasks. By integrating code semantics, repository structure, and call graph dependencies, CoRet significantly improves retrieval recall, a crucial step for efficient code modification.
What Problem Does This Paper Address?
Code editing, whether by humans or AI, requires efficiently locating the right pieces of code within a potentially massive codebase. Existing methods often struggle with this "repository-level retrieval" because they fail to capture the full context: the code's meaning (semantics), its organizational structure (hierarchy), and how different parts interact (dependencies). This makes it hard to find relevant code for tasks like implementing new features or fixing bugs, especially in large, real-world projects. The paper argues that standard pre-trained models, while good at understanding code semantics, miss the crucial repository-level context needed for effective code editing.
Methods
CoRet is a dense retrieval model that tackles this problem by:
- Chunking: Breaking down code files into smaller, manageable units called "chunks" (functions, classes, methods) to provide finer-grained retrieval.
- Repository-Level Loss Function: Introducing a new loss function specifically designed for repository-level retrieval. This loss function aims to maximize the similarity between a natural language query and the correct code chunks, while minimizing similarity to irrelevant ones. It's framed as a maximum likelihood estimation problem.
- Contextual Integration: Incorporating two key types of context:
- Repository Hierarchy: Including the file path as part of the chunk representation.
- Call Graph Dependencies: Leveraging information about which code chunks invoke or are invoked by others. This is done by concatenating neighboring chunks in the call graph with the target chunk.
- Model Architecture: CoRet uses a modified CodeSage model as its backbone, with specific adaptations for the retrieval task.
Datasets
- Training: The model is trained on the SWE-bench dataset, which provides real-world GitHub repositories and associated bug-fixing tasks.
- Evaluation: CoRet is evaluated on two datasets:
- SWE-bench Verified: A subset of SWE-bench.
- Long Code Arena (LCA): A bug localization dataset.
Key Insights and Ideas
- Repository-Level Retrieval is Different: Standard contrastive learning methods, effective for general representation learning, are suboptimal for the specific task of repository-level code editing retrieval.
- Context is King: Simply relying on code semantics isn't enough. Integrating repository structure (file paths) and dynamic relationships (call graphs) provides a significant boost in retrieval performance.
- A Dedicated Loss Function Matters: A loss function tailored to repository-level retrieval, rather than generic contrastive losses, leads to substantial improvements.
- Fine-Grained Chunks are Crucial: Breaking down code into functions, classes, and methods (chunks) is more effective than treating entire files as retrieval units.
Key Results
CoRet demonstrates significant improvements over existing models:
- Recall Improvement: CoRet achieves at least a 15 percentage point improvement in retrieval recall on both SWE-bench and Long Code Arena datasets compared to baseline methods.
- Perfect Recall: On the SWE-bench Verified dataset, CoRet achieves a perfect recall of 0.53 (chunk level) and 0.47 (file level), significantly outperforming the best baseline (CodeSage S) which achieved 0.35 and 0.28 respectively.
- Call Graph Impact: Including call graph context improves multi-chunk retrieval, as shown by comparing CoRet-CG with the full CoRet model.
- File Path Importance: Including file paths in chunk representation is shown to be important, with a performance drop when removed.
Limitations and Future Work
- Language Specificity: The current experiments are limited to Python code.
- Dataset Scope: Experiments were conducted on SWE-bench and LCA, which might have limitations (e.g., SWE-bench often involves edits to a single file).
- Encoder-Only Focus: The investigation was limited to encoder models; modern LLMs might offer better baseline performance but require more resources.
- Future Directions:
- Extending to multi-language datasets (e.g., SWE-PolyBench).
- Exploring better strategies for selecting relevant call graph neighbors, potentially leveraging topological properties.
- Investigating LLMs for retrieval, as they might provide better baseline performance.
Summary updated