TL;DR
This paper introduces SWE-Refactor, a novel benchmark for evaluating Large Language Models (LLMs) on code refactoring tasks. It addresses limitations of existing benchmarks by focusing on real-world, developer-written refactorings from Java projects, including complex, multi-file changes, and providing rich repository-level context. The benchmark reveals that while LLMs excel at simple refactorings, they struggle with compound and cross-file operations, highlighting areas for future research and development.
What Problem Does This Paper Address?
Code refactoring is a crucial software engineering practice for improving code quality, maintainability, and adaptability. While LLMs have shown promise in code generation, their application to refactoring is more challenging due to the need for precise, behavior-preserving edits and understanding complex code dependencies within a repository. Existing benchmarks for LLM-based refactoring often suffer from:
- Limited scope: Focusing on atomic refactorings and lacking real-world complexity.
- Noisy data: Including unrelated code changes alongside refactorings, making evaluation ambiguous.
- Lack of repository context: Missing crucial information like class hierarchies and method call graphs.
- Manual construction: Requiring significant human effort, limiting scalability and reproducibility.
SWE-Refactor aims to bridge this gap by providing a comprehensive, repository-level benchmark that captures the nuances of real-world code refactoring.
Methods
The authors developed SWE-Refactor using a four-step automated pipeline:
- Mine Refactorings via Static Analysis: They leveraged Abstract Syntax Tree (AST)-based refactoring detection tools (like RefactoringMiner) to extract commits containing refactorings from GitHub repositories. This step focused on identifying refactoring types and their locations within files.
- Curate Pure and Targeted Refactoring Types: Using AST analysis and a tool called PurityChecker, they filtered out commits with non-refactoring changes (e.g., bug fixes) and retained only six specific refactoring types (atomic and compound).
- Enrich Refactoring Changes with Multi-Level Code Information: They used the Eclipse Java Development Tools (JDT) to gather rich repository-level context, including project structure, class hierarchies, method caller/callee relationships, and build configurations.
- Verify Compilation and Test Coverage: A script was developed to compile projects and run test suites using JaCoCo to ensure behavioral equivalence and that refactored code was exercised by tests.
For evaluation, they employed three metrics: compilation and test success (functional verification), AST-based refactoring verification (ensuring the correct refactoring was applied), and CodeBLEU (measuring human-likeness).
Datasets
The primary dataset introduced is SWE-Refactor, comprising 1,099 developer-written, behavior-preserving refactorings mined from 18 diverse Java projects. This dataset includes 922 atomic and 177 compound instances.
The authors also utilized the Refactoring Oracle Dataset (Tsantalis et al., 2020) for constructing retrieval-augmented generation (RAG) prompts, specifically extracting 905 pure method-level refactorings for this purpose.
Key Insights and Ideas
- Real-world complexity matters: SWE-Refactor's focus on developer-written, repository-level refactorings reveals that LLMs struggle significantly with compound and cross-file operations, which are common in practice.
- Atomic vs. Compound refactorings are a key differentiator: LLMs perform much better on simpler, single-method refactorings (like Extract Method) than on multi-step, multi-file transformations.
- Context is king (and hard): Providing rich repository-level context (class structure, call graphs) is crucial for LLMs to perform refactorings accurately, but even with this context, LLMs can fail to maintain cross-file consistency.
- Multi-agent workflows show promise: An iterative approach with a Developer Agent and a Reviewer Agent demonstrated improved performance, especially on complex refactorings, by enabling refinement through feedback.
- Java is an important target: The choice of Java for the benchmark broadens evaluation beyond the Python-centric landscape of many existing benchmarks, reflecting its prevalence in enterprise systems.
Key Results
- SWE-Refactor contains 1,099 pure refactorings from 18 Java projects.
- General-purpose LLMs (like GPT-4o-mini and DeepSeek-V3) generally outperformed open-source code LLMs.
- DeepSeek-V3 achieved the highest overall success rate (41.58%), followed closely by GPT-4o-mini (39.85%).
- GPT-5.1-Codex achieved only 39.4% success on compound refactorings, highlighting a significant challenge.
- Multi-agent workflows (using GPT-4o-mini) achieved higher overall success (579 refactorings) compared to simple prompting (438) and RAG (451), particularly for compound refactorings.
Limitations and Future Work
- Language limitation: SWE-Refactor currently focuses solely on Java. The authors plan to extend it to other languages to support multi-language evaluation.
- Scope of refactorings: The benchmark primarily targets method-level refactorings. Higher-level (e.g., class-level) refactorings are less frequent and harder to isolate, but the authors aim to include a broader range in the future.
- Dataset scale: While 1,099 instances is substantial, the authors acknowledge that a larger dataset would be beneficial for comprehensive evaluation and LLM fine-tuning. They plan to expand the dataset.
- Future work: The authors suggest further research into improving LLM performance on compound refactorings, exploring more sophisticated agentic workflows, and developing better evaluation metrics that capture nuanced aspects of code quality beyond functional correctness.
Summary updated