RETRACE: From Entangled Repair Histories to Reusable Experience for CI Repair

Rabeya Khatun Muna, Muhammad Ahasanuzzaman, Nakhla Rafi, Yisen Xu, Jinqiu Yang, Tse-Hsun ChenarXiv 2026 · arXiv

TL;DR

RETRACE is a new framework that transforms messy, "entangled" software pull request histories into clean, reusable repair experience for AI agents. By reconciling the final patch with the step-by-step commit history, it allows LLMs to fix complex Continuous Integration (CI) failures with significantly higher accuracy, improving Pass@1 rates by over 10 percentage points on challenging benchmarks.

What Problem Does This Paper Address?

Modern software development relies on Continuous Integration (CI) to catch bugs early. When a CI build fails, developers submit Pull Requests (PRs) to fix it. While these PRs are a goldmine of "experience," they are incredibly noisy. A single PR might contain multiple commits that fix three different bugs, revert two failed attempts, and include unrelated feature work.

For a Large Language Model (LLM) agent, this "entangled" history is difficult to learn from. Most existing methods assume that a problem (like a bug report) is already perfectly aligned with its solution (a clean patch). In the real world, this alignment doesn't exist. The authors tackle the problem–repair attribution task: how do we look at a messy history and figure out exactly which change fixed which specific CI failure?

Methods

The authors introduce RETRACE (Repair experience through Change Attribution in Pull requests), which reconstructs problem-level experience using a multi-view approach:

  1. Multi-View Reconstruction: It looks at the history from two angles. The Endpoint View reasons backward from the final successful patch to see what changes actually "stuck." The Development View traces the commit history forward to see how the repair evolved.
  2. Evidence Reconciliation: RETRACE uses CI execution logs and workflow definitions to act as the "ground truth," linking specific code changes to the failures they were intended to resolve.
  3. Three Levels of Abstraction: To make the experience reusable across different projects, RETRACE stores knowledge at three levels:
  4. L1 (Concrete): The exact code changes.
  5. L2 (Contextual Strategy): A description of how to recognize and fix similar failures within the same repo.
  6. L3 (Transferable Pattern): A generalized repair principle that can apply to entirely different repositories.
  7. Dynamic Reuse: When faced with a new failure, RETRACE decomposes it into "atomic problems," retrieves the most relevant L1-L3 experiences, and guides a repair agent through a sequential fix process.
Figure 1: Overview of RETRACE.

Datasets

The study uses CI-REPAIR-BENCH, a benchmark containing 565 PR-level repairs from 101 different repositories.

As shown in the table below, CI-REPAIR-BENCH is significantly more complex than standard benchmarks like SWE-bench. The patches are much larger, modifying an average of 10.5 files compared to SWE-bench's 1.7. Furthermore, over 83% of the instances in this benchmark involve multiple concurrent failure types, making "entanglement" a primary challenge.

Table 1: Comparison of change size across repository-level software repair benchmarks.

Key Insights and Ideas

  • Reconciliation is Key: The authors found that using only the final patch (endpoint) or only the commit history (development) is insufficient. Combining both views consistently yields the best results because it filters out the noise of "failed attempts" while retaining the "why" found in the history.
  • Abstraction Enables Transfer: By abstracting repairs into L3 "Transferable Patterns," the system can use a fix from a Python web framework to help repair a Javascript utility library if the underlying failure mechanism (e.g., a dependency mismatch) is the same.
  • Sequential Problem Solving: CI failures often hide downstream problems. RETRACE's ability to identify "hidden" or "recurring" problems from history allows it to prepare the agent for failures that haven't even appeared in the logs yet.

Key Results

RETRACE demonstrated substantial improvements across multiple LLM backbones (MiniMax-M2.5, DeepSeek-V4-Flash, and Codex).

  • Accuracy Boost: For the mini-SWE-agent using MiniMax-M2.5, RETRACE improved the Pass@1 rate from 19.6% to 31.9%.
  • Outperforming Baselines: It significantly beat Historical-Plan (based on the ThinkRepair method), which actually performed worse than having no experience at all. This suggests that simply providing raw historical plans adds too much noise; structured reconstruction is essential.
  • Cost Efficiency: Interestingly, for the Codex model, RETRACE not only improved accuracy but also reduced the average inference cost from $0.142 to $0.118 per instance, as the retrieved experience helped the agent avoid unnecessary exploration.
Table 2: Main results on CI-REPAIR-BENCH. ∗indicates that RETRACE significantly outperforms the marked baseline (exact McNemar test with Holm correction; pHolm < 0.05).

Limitations and Future Work

  • Incomplete Histories: If a developer squashes their commits or the history is otherwise unavailable, RETRACE must rely solely on the endpoint view, which is less informative.
  • Attribution Complexity: In extremely large PRs with hundreds of changed lines, even LLM-based attribution can struggle to perfectly separate concerns.
  • Validation Overhead: Currently, RETRACE relies on lightweight local validation between repair steps because running a full CI suite for every minor change is too slow and expensive. Improving the speed of "full-fidelity" feedback is a clear next step.

Summary updated

Abstract

Large language model (LLM) agents increasingly reuse prior experience, but most approaches assume that problems and solutions are already aligned. Software histories rarely provide this alignment: a pull request (PR) may contain multiple continuous integration (CI) problems, failed attempts, reverted edits, and unrelated changes, obscuring which changes resolve each problem. We present RETRACE, a framework for reconstructing problem-level repair experience from such histories. RETRACE combines an endpoint view that reasons backward from changes retained in the passing revision with a development view that traces repair evolution forward through commit history. CI execution evidence reconciles the two views, and the recovered experience is represented at three abstraction levels, from concrete fixes to transferable repair patterns. For new failures, RETRACE retrieves relevant problem-level experience to guide repair. On CI-REPAIR-BENCH, comprising 565 PR-level repairs from 101 repositories across 12 failure categories, RETRACE improves mini-SWE-agent Pass@1 from 19.6% to 31.9% with MiniMax-M2.5 and from 23.3% to 32.8% with DeepSeek-V4-Flash. On a matched subset, Codex improves from 15.5% to 27.5%. Combining both views consistently outperforms either alone, showing that recovering problem-change alignment enables historical CI repairs to serve as reusable repair experience.
BibTeX
@misc{muna2026retrace,
  author = {Rabeya Khatun Muna and Muhammad Ahasanuzzaman and Nakhla Rafi and Yisen Xu and Jinqiu Yang and Tse-Hsun Chen},
  title = {{RETRACE: From Entangled Repair Histories to Reusable Experience for CI Repair}},
  year = {2026},
  eprint = {2610.04658},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.04658},
}
Figure