TL;DR
Coding agents can now "self-evolve" by reflecting on past tasks to update their own operating environment, or "harness." This study finds that while agents frequently propose experience-driven improvements, these adaptations often fail to generalize to new tasks and can even introduce unnecessary overhead or new errors.
What Problem Does This Paper Address?
Modern coding agents like Claude Code don't just write code; they operate within a harness—a collection of configuration files (like CLAUDE.md), environment settings, and tool instructions. A promising frontier in AI engineering is "harness self-evolution," where an agent performs a retrospective after a task to update its harness, theoretically becoming more efficient at future tasks in the same repository.
However, it remains unclear if these self-proposed changes actually help. Does a fix for a specific test failure in one session prevent "repeated work" in the next? Or does the agent simply overfit its environment to a single task's quirks? This paper empirically investigates the quality and utility of these retrospective adaptations.
Methods
The researchers conducted a multi-stage empirical study using Claude Code on SWE-bench Verified issues:
- Generation: They ran 30 task sessions and invoked the
/retroskill to generate 125 harness change proposals. - Qualitative Coding: Two LLMs (Claude and GPT) and the authors categorized these proposals based on their Trigger (was it based on a session event or just a general "best practice"?) and Mechanism (what kind of change was it?).
- Implementation & Transfer: They implemented 24 proposals and evaluated 12 of them on different issues within the same repositories.
- Evaluation: They measured recurrence—whether the agent repeated the specific inefficient action the adaptation was meant to prevent (e.g., trying to run a non-existent test runner).
Datasets
The study utilized SWE-bench Verified, a benchmark of real-world GitHub issues.
- Exploratory Sample: 10 issues sampled from Django, SymPy, Matplotlib, Astropy, and Sphinx.
- Targeted Sample: 20 additional issues from Astropy, pytest, Sphinx, and SymPy, specifically chosen because they were likely to reproduce environment-related problems encountered in the first phase.
Key Insights and Ideas
- Retrospectives are more than just logs: Agents don't just translate events into rules. They often recommend "prior-driven" changes—practices the agent considers "good" regardless of whether their absence caused a problem (e.g., suggesting a linting hook even if no linting errors occurred).
- The Generalization Gap: Adaptations that are too specific (like a list of three specific failing tests) often fail when the next task encounters a fourth failing test. The agent then reverts to its original, inefficient behavior (like running full baseline comparisons).
- Adaptations can be "Double-Edged Swords": An adaptation intended to save time can actually add work. For example, adding a instruction to run CI lint checks before finishing might prevent errors but significantly increases the session's elapsed time and cost.
- Self-Evolution requires Pruning: The authors argue that a truly autonomous self-evolving system needs a way to verify if a change actually helps on later tasks and a mechanism to "garbage collect" or revise ineffective instructions.
Key Results
The researchers categorized the 125 proposals into several mechanisms, as shown in the table below:
- RQ1 (Proposals): 101 of 125 proposals were "experience-driven." However, 18 of the 19 "prior-driven" proposals were related to linting, showing the agent's bias toward standardizing environment quality.
- RQ2 (Effectiveness): The results were mixed. In "informative" transfer events (where the problem actually reappeared):
- 2 adaptations successfully eliminated the targeted repeated work.
- 1 adaptation reduced it.
- 4 adaptations showed no reduction at all.
As seen in the transfer event data, even when the agent "consulted" the adaptation, it didn't always change its behavior effectively:
Qualitatively, one Sphinx adaptation actually caused the agent to perform more work on a control task, increasing median elapsed time from 82 to 245 seconds.
Limitations and Future Work
The study acknowledges several limitations:
- Sample Size: The evaluation focused on 12 specific adaptations across 13 transfer events.
- Human-in-the-loop: The researchers selected which proposals to implement, meaning the process wasn't a fully autonomous loop.
- Scope: The study focused heavily on environment and test-runner problems.
Future Work: The authors suggest evaluating how agents handle adaptations for code navigation and coding conventions, and developing systems that can autonomously evaluate and refine their own harness changes over time.
Summary updated