TL;DR
RETRACE is a new framework that transforms messy, "entangled" software pull request histories into clean, reusable repair experience for AI agents. By reconciling the final patch with the step-by-step commit history, it allows LLMs to fix complex Continuous Integration (CI) failures with significantly higher accuracy, improving Pass@1 rates by over 10 percentage points on challenging benchmarks.
What Problem Does This Paper Address?
Modern software development relies on Continuous Integration (CI) to catch bugs early. When a CI build fails, developers submit Pull Requests (PRs) to fix it. While these PRs are a goldmine of "experience," they are incredibly noisy. A single PR might contain multiple commits that fix three different bugs, revert two failed attempts, and include unrelated feature work.
For a Large Language Model (LLM) agent, this "entangled" history is difficult to learn from. Most existing methods assume that a problem (like a bug report) is already perfectly aligned with its solution (a clean patch). In the real world, this alignment doesn't exist. The authors tackle the problem–repair attribution task: how do we look at a messy history and figure out exactly which change fixed which specific CI failure?
Methods
The authors introduce RETRACE (Repair experience through Change Attribution in Pull requests), which reconstructs problem-level experience using a multi-view approach:
- Multi-View Reconstruction: It looks at the history from two angles. The Endpoint View reasons backward from the final successful patch to see what changes actually "stuck." The Development View traces the commit history forward to see how the repair evolved.
- Evidence Reconciliation: RETRACE uses CI execution logs and workflow definitions to act as the "ground truth," linking specific code changes to the failures they were intended to resolve.
- Three Levels of Abstraction: To make the experience reusable across different projects, RETRACE stores knowledge at three levels:
- L1 (Concrete): The exact code changes.
- L2 (Contextual Strategy): A description of how to recognize and fix similar failures within the same repo.
- L3 (Transferable Pattern): A generalized repair principle that can apply to entirely different repositories.
- Dynamic Reuse: When faced with a new failure, RETRACE decomposes it into "atomic problems," retrieves the most relevant L1-L3 experiences, and guides a repair agent through a sequential fix process.
Datasets
The study uses CI-REPAIR-BENCH, a benchmark containing 565 PR-level repairs from 101 different repositories.
As shown in the table below, CI-REPAIR-BENCH is significantly more complex than standard benchmarks like SWE-bench. The patches are much larger, modifying an average of 10.5 files compared to SWE-bench's 1.7. Furthermore, over 83% of the instances in this benchmark involve multiple concurrent failure types, making "entanglement" a primary challenge.
Key Insights and Ideas
- Reconciliation is Key: The authors found that using only the final patch (endpoint) or only the commit history (development) is insufficient. Combining both views consistently yields the best results because it filters out the noise of "failed attempts" while retaining the "why" found in the history.
- Abstraction Enables Transfer: By abstracting repairs into L3 "Transferable Patterns," the system can use a fix from a Python web framework to help repair a Javascript utility library if the underlying failure mechanism (e.g., a dependency mismatch) is the same.
- Sequential Problem Solving: CI failures often hide downstream problems. RETRACE's ability to identify "hidden" or "recurring" problems from history allows it to prepare the agent for failures that haven't even appeared in the logs yet.
Key Results
RETRACE demonstrated substantial improvements across multiple LLM backbones (MiniMax-M2.5, DeepSeek-V4-Flash, and Codex).
- Accuracy Boost: For the
mini-SWE-agentusing MiniMax-M2.5, RETRACE improved the Pass@1 rate from 19.6% to 31.9%. - Outperforming Baselines: It significantly beat
Historical-Plan(based on the ThinkRepair method), which actually performed worse than having no experience at all. This suggests that simply providing raw historical plans adds too much noise; structured reconstruction is essential. - Cost Efficiency: Interestingly, for the Codex model, RETRACE not only improved accuracy but also reduced the average inference cost from $0.142 to $0.118 per instance, as the retrieved experience helped the agent avoid unnecessary exploration.
Limitations and Future Work
- Incomplete Histories: If a developer squashes their commits or the history is otherwise unavailable, RETRACE must rely solely on the endpoint view, which is less informative.
- Attribution Complexity: In extremely large PRs with hundreds of changed lines, even LLM-based attribution can struggle to perfectly separate concerns.
- Validation Overhead: Currently, RETRACE relies on lightweight local validation between repair steps because running a full CI suite for every minor change is too slow and expensive. Improving the speed of "full-fidelity" feedback is a clear next step.
Summary updated