How Do Coding-Agent Harnesses Self-Evolve? An Empirical Study of Retrospective Adaptation

Christoph Treude, Sebastian BaltesarXiv 2026 · arXiv

TL;DR

Coding agents can now "self-evolve" by reflecting on past tasks to update their own operating environment, or "harness." This study finds that while agents frequently propose experience-driven improvements, these adaptations often fail to generalize to new tasks and can even introduce unnecessary overhead or new errors.

What Problem Does This Paper Address?

Modern coding agents like Claude Code don't just write code; they operate within a harness—a collection of configuration files (like CLAUDE.md), environment settings, and tool instructions. A promising frontier in AI engineering is "harness self-evolution," where an agent performs a retrospective after a task to update its harness, theoretically becoming more efficient at future tasks in the same repository.

However, it remains unclear if these self-proposed changes actually help. Does a fix for a specific test failure in one session prevent "repeated work" in the next? Or does the agent simply overfit its environment to a single task's quirks? This paper empirically investigates the quality and utility of these retrospective adaptations.

Methods

The researchers conducted a multi-stage empirical study using Claude Code on SWE-bench Verified issues:

  1. Generation: They ran 30 task sessions and invoked the /retro skill to generate 125 harness change proposals.
  2. Qualitative Coding: Two LLMs (Claude and GPT) and the authors categorized these proposals based on their Trigger (was it based on a session event or just a general "best practice"?) and Mechanism (what kind of change was it?).
  3. Implementation & Transfer: They implemented 24 proposals and evaluated 12 of them on different issues within the same repositories.
  4. Evaluation: They measured recurrence—whether the agent repeated the specific inefficient action the adaptation was meant to prevent (e.g., trying to run a non-existent test runner).

Datasets

The study utilized SWE-bench Verified, a benchmark of real-world GitHub issues.

  • Exploratory Sample: 10 issues sampled from Django, SymPy, Matplotlib, Astropy, and Sphinx.
  • Targeted Sample: 20 additional issues from Astropy, pytest, Sphinx, and SymPy, specifically chosen because they were likely to reproduce environment-related problems encountered in the first phase.

Key Insights and Ideas

  • Retrospectives are more than just logs: Agents don't just translate events into rules. They often recommend "prior-driven" changes—practices the agent considers "good" regardless of whether their absence caused a problem (e.g., suggesting a linting hook even if no linting errors occurred).
  • The Generalization Gap: Adaptations that are too specific (like a list of three specific failing tests) often fail when the next task encounters a fourth failing test. The agent then reverts to its original, inefficient behavior (like running full baseline comparisons).
  • Adaptations can be "Double-Edged Swords": An adaptation intended to save time can actually add work. For example, adding a instruction to run CI lint checks before finishing might prevent errors but significantly increases the session's elapsed time and cost.
  • Self-Evolution requires Pruning: The authors argue that a truly autonomous self-evolving system needs a way to verify if a change actually helps on later tasks and a mechanism to "garbage collect" or revise ineffective instructions.

Key Results

The researchers categorized the 125 proposals into several mechanisms, as shown in the table below:

Mechanism codes of the 125 proposals, showing that executable checks and navigation pointers were the most common.
  • RQ1 (Proposals): 101 of 125 proposals were "experience-driven." However, 18 of the 19 "prior-driven" proposals were related to linting, showing the agent's bias toward standardizing environment quality.
  • RQ2 (Effectiveness): The results were mixed. In "informative" transfer events (where the problem actually reappeared):
  • 2 adaptations successfully eliminated the targeted repeated work.
  • 1 adaptation reduced it.
  • 4 adaptations showed no reduction at all.

As seen in the transfer event data, even when the agent "consulted" the adaptation, it didn't always change its behavior effectively:

Table showing informative transfer events. Only two adaptations (SymPy and Astropy) successfully eliminated recurrence (0/5).

Qualitatively, one Sphinx adaptation actually caused the agent to perform more work on a control task, increasing median elapsed time from 82 to 245 seconds.

Limitations and Future Work

The study acknowledges several limitations:

  • Sample Size: The evaluation focused on 12 specific adaptations across 13 transfer events.
  • Human-in-the-loop: The researchers selected which proposals to implement, meaning the process wasn't a fully autonomous loop.
  • Scope: The study focused heavily on environment and test-runner problems.

Future Work: The authors suggest evaluating how agents handle adaptations for code navigation and coding conventions, and developing systems that can autonomously evaluate and refine their own harness changes over time.

Summary updated

Abstract

Coding agents can adapt their harnesses, the software, configuration, and instructions used to run them, based on experience from earlier tasks, but whether these adaptations improve their behavior on later tasks remains unclear. We study \retro, a skill that asks an agent to review a completed session and propose harness changes, and ask what changes it proposes and whether implemented changes reduce repeated work on later tasks. We ran Claude Code on 30 SWE-bench Verified issues, qualitatively coded the 125 proposals from \retro, implemented 24, and evaluated 12 harness changes on later issues from the same repositories in fresh sessions with and without each change. Most proposals referred to an event in the session, but 18 of 43 lint-related proposals recommended a practice regardless of what had happened. Among the seven evaluations in which sessions without the change encountered the targeted problem, two changes eliminated the targeted action, one reduced its occurrence, four showed no reduction, and some changes added work on unrelated issues. Harness self-evolution requires more than generating and implementing changes: it needs ways to establish whether a change helps on later tasks and to revise or remove changes that do not.
BibTeX
@misc{treude2026how,
  author = {Christoph Treude and Sebastian Baltes},
  title = {{How Do Coding-Agent Harnesses Self-Evolve? An Empirical Study of Retrospective Adaptation}},
  year = {2026},
  eprint = {2610.34938},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34938},
}
Figure