TL;DR
This paper reveals that current code retrieval systems heavily rely on superficial textual features like docstrings, leading to a bias towards well-documented but irrelevant code. To combat this, they propose SACL, a framework that augments code retrieval with semantic descriptions, significantly improving both retrieval and downstream code generation performance.
What Problem Does This Paper Address?
Retrieval-Augmented Code Generation (RACG) is a powerful technique for generating code by first retrieving relevant code snippets. However, the effectiveness of RACG hinges on the quality of the retrieval system. This paper identifies a critical bottleneck: current code retrievers, despite being trained on code, exhibit a strong bias towards surface-level textual features (like docstrings and identifier names) rather than the underlying functional semantics of the code. This bias leads them to favor well-documented but irrelevant code over functionally correct but poorly documented code, hindering accurate retrieval and, consequently, code generation.
Methods
The authors propose SACL (Semantic-Augmented Reranking and Localization), a framework designed to address the identified textual bias. SACL introduces two main components:
- Semantic-Augmented Code Reranking: After initial retrieval, SACL uses a Large Language Model (LLM) to generate concise, natural language descriptions of the retrieved code snippets. The final ranking score is a weighted combination of the original code-query similarity and the description-query similarity. This transforms the cross-modal (text-to-code) comparison into a more tractable text-to-text comparison, emphasizing semantic meaning.
- Semantic-Augmented In-context Localization: For repository-level code generation, SACL enhances the representation of the repository structure. It augments file names and directory structures with LLM-generated semantic descriptions of each file's contents, purpose, and relationships to other files. This richer context helps LLMs better understand the codebase and localize relevant files for modification tasks.
Datasets
The paper evaluates SACL on three widely-used benchmarks:
- HumanEval: A dataset of 164 hand-written programming problems with function signatures and test cases.
- MBPP (Mostly Basic Python Problems): A dataset of 974 Python programming tasks with natural language descriptions and test cases.
- SWE-Bench-Lite: A repository-level issue-solving dataset containing 300 real-world GitHub tasks.
Key Insights and Ideas
- Textual Bias is Real and Significant: Code retrievers are surprisingly sensitive to superficial textual cues like docstrings and identifier names, often ignoring the actual code functionality.
- Well-Documented ≠ Relevant: The systems exhibit a strong bias towards code with good documentation, even if that documentation is irrelevant to the query. This can lead to incorrect retrieval.
- LLMs as Semantic Bridges: LLMs can effectively generate semantic descriptions of code that capture functional intent, even when surface-level textual features are normalized. These descriptions act as a crucial bridge between natural language queries and code semantics.
- Augmenting Structure with Semantics: For repository-level tasks, simply providing file structure isn't enough. Augmenting this structure with semantic descriptions of file contents significantly improves localization accuracy.
Key Results
- Code Retrieval Improvement: SACL substantially improves code retrieval performance, achieving up to 12.8% Recall@1 gain on HumanEval and 9.4% on MBPP under challenging normalization settings.
- Code Generation Improvement: The enhanced retrieval directly translates to better code generation, with SACL leading to up to 4.88% Pass@1 gain on HumanEval and 1.67% on SWE-Bench-Lite.
- Fault Localization: On SWE-Bench-Lite, SACL improves file localization accuracy by up to 8.0%.
- Bias Mitigation: Experiments show that SACL effectively mitigates the bias towards well-documented code, even when the ground truth code is poorly documented.
Limitations and Future Work
- Agentic Methods: The current semantic augmentation techniques are not directly applied to agentic methods for code generation.
- Benchmark Scope: The analysis of lexical bias primarily focuses on function retrieval and could be extended to other repository-level coding benchmarks.
- Future Work: The authors suggest exploring the application of their semantic augmentation techniques to agentic code generation and expanding the analysis to a wider range of repository-level coding tasks. They also highlight the potential for further research into understanding and mitigating other forms of bias in code retrieval.
Summary updated