MANTRA: Enhancing Automated Method-Level Refactoring with Contextual RAG and Multi-Agent LLM Collaboration

TL;DR

This paper introduces MANTRA, an LLM-powered framework that automates method-level code refactoring. By combining retrieval-augmented generation, multi-agent collaboration, and self-repair mechanisms, MANTRA significantly improves the success rate of generating compilable and test-passing refactored code compared to existing LLM approaches and even surpasses IntelliJ's refactoring tool.

What Problem Does This Paper Address?

Software maintenance and evolution heavily rely on code refactoring – restructuring code without changing its external behavior to improve its design and readability. However, manual refactoring is time-consuming and error-prone, often leading to the introduction of new bugs. While Large Language Models (LLMs) show promise in automating this process, current LLM-based solutions struggle with ensuring code compilability, passing tests, and handling a broad range of refactoring types. They often lack the sophisticated reasoning and verification steps that human developers employ, leading to limited effectiveness and acceptance.

Methods

MANTRA tackles these challenges with a three-pronged approach:

1. Context-Aware Retrieval-Augmented Generation (RAG): MANTRA builds a database of "pure refactorings" (changes solely for structural improvement) and enhances it with natural language descriptions and caller-callee relationships. This allows it to retrieve relevant examples to guide the LLM, improving the quality and contextuality of generated refactorings.
2. Multi-Agent Collaboration: MANTRA simulates a human refactoring process using three agents:
3. Developer Agent: Analyzes code, retrieves examples via RAG, and generates the initial refactored code.
4. Reviewer Agent: Verifies refactoring correctness, code style (using tools like CheckStyle), and then triggers compilation and testing.
5. Repair Agent: If compilation or tests fail, this agent uses verbal reinforcement learning (inspired by the Reflexion framework) to iteratively identify and fix issues.
6. Self-Repair Using Verbal Reinforcement Learning: This component allows the system to learn from errors. The Repair Agent analyzes failures, reflects on the causes, plans corrections, and acts on them, repeating until the code is functional or a limit is reached.

Datasets

The study uses a dataset of 703 "pure refactorings" extracted from 10 representative Java projects. These projects were chosen for their diverse domains, rich commit histories, and availability of tests. The "pure refactorings" were filtered using PurityChecker to exclude commits with unrelated changes (like bug fixes or feature additions), ensuring a clean evaluation of refactoring effectiveness.

Key Insights and Ideas

• Multi-agent collaboration is key: Simulating the developer-reviewer-repair cycle significantly improves refactoring quality and correctness.
• Context matters: Integrating natural language descriptions and caller-callee relationships into the RAG process provides crucial context for better refactoring.
• Verification is non-negotiable: Relying solely on LLM generation without robust compilation, testing, and refactoring verification (using tools like RefactoringMiner) leads to a high failure rate.
• LLM choice impacts performance: The paper notes a performance drop when switching from GPT-4o-mini to GPT-3.5-turbo, highlighting the importance of the underlying LLM.
• Human-like refactoring: MANTRA's generated code is perceived by developers as having similar readability and reusability to human-written code, and in some cases, even better due to detailed comments and descriptive naming.

Key Results

• High Success Rate: MANTRA achieved an 82.8% success rate (582/703) in generating compilable and test-passing refactored code, a massive improvement over the baseline RawGPT's 8.7% (61/703).
• Outperforms IDE Tools: Compared to IntelliJ's EM-Assist for Extract Method refactoring, MANTRA showed a 50% improvement in success rate.
• Code Similarity: MANTRA's generated code showed higher CodeBLEU scores and AST Precision/Recall compared to RawGPT, indicating greater similarity to human refactorings.
• User Study Findings: 37 developers found MANTRA-generated code to be as readable and reusable as human code, with a preference for MANTRA in specific refactoring types like Extract & Move and Move & Rename due to better comments and naming.
• Component Contributions: An ablation study revealed that removing any of MANTRA's core components (RAG, Reviewer Agent, Repair Agent) significantly reduced its performance, with the Reviewer Agent showing the most substantial impact.

Limitations and Future Work

• LLM Variability: The generative nature of LLMs can lead to response variations. The authors mitigated this by setting the temperature to 0, but further research on LLM impact is suggested.
• Scope of Refactoring: The study focused on method-level refactorings in Java. Future work should explore MANTRA's effectiveness on other refactoring types (e.g., class-level) and in different programming languages.
• Metric Limitations: While CodeBLEU and AST metrics are useful, they might not capture all nuances of code similarity. The user study helps address this, but subjective elements remain.
• Future Directions: The authors suggest exploring MANTRA's applicability to broader refactoring scenarios and other languages, and further investigating the synergy between traditional software engineering tools and LLM-based approaches.

@misc{xu2025mantra,
  author = {Yisen Xu and Feng Lin and Jinqiu Yang and Tse-Hsun Chen and Nikolaos Tsantalis},
  title = {{MANTRA: Enhancing Automated Method-Level Refactoring with Contextual RAG and Multi-Agent LLM Collaboration}},
  year = {2025},
  eprint = {2503.14340},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.14340},
}