Agent Self-Improvement with Frozen Weights: The Added Value and Limits of Failure Traces

Arkid Mitra

TL;DR

This paper investigates whether AI agents can truly improve by analyzing their own "failure traces" (records of past mistakes). While giving a teacher model access to these traces does provide a modest boost in isolated decision-making, about 60% of the total improvement comes from simple prompt optimization that never even sees the agent's errors.

What Problem Does This Paper Address?

The AI community is currently fascinated by "self-evolving" agents—systems that supposedly get better over time by reflecting on their own performance. However, most of these agents have "frozen weights" (their core neural network doesn't change), meaning any "learning" must happen in the "harness": the system prompts, notes, and control logic surrounding the model.

The authors tackle a critical question: Is the gain we see in these systems actually coming from the content of the failures, or is it just the result of a better search for effective prompts? If a teacher model can improve an agent just as well without seeing its mistakes, then "learning from experience" might be an overstatement.

Methods

The researchers used a Qwen3.5-9B model as their agent and a larger "teacher" model to write "lessons" (formatted as WHEN situation THEN action) to guide it.

The core innovation is a rigorous experimental design featuring:

  • No-Evidence Search (The Control): A teacher model tries to improve the agent's prompt using the same budget and feedback loop, but without seeing any of the agent's actual interaction traces. This isolates the "added value" of the failure data.
  • The Gate: A hierarchical Bayesian screening process that only admits lessons if they show a credible gain across multiple contexts without causing "harm" (regressions) elsewhere.
  • Blind Scorer Audit: To ensure results weren't skewed by benchmark errors, independent reviewers audited the ground truth of the test sets.

Datasets

The study evaluated agents on two primary benchmarks:

  1. $\tau^2$-bench retail: A customer-service domain with complex policies. It was tested at the "decision level" (isolated choices) and the "conversation level" (full multi-turn interactions).
  2. BFCL (Berkeley Function Calling Leaderboard) multi-turn: A benchmark for tool-use and API calling.

Key Insights and Ideas

  • Search is the Silent Winner: A significant portion of what looks like "learning from experience" is actually just the teacher model finding better ways to phrase existing rules. In their retail tests, search alone achieved 60% of the total improvement.
  • The Generalization Gap: Lessons that help an agent make a single correct decision often fail to improve—or even hurt—performance in full, fluid conversations.
  • Benchmarks are Brittle: The authors found that a staggering 60.8% of the held-out instances in the BFCL benchmark were "defective" (where a correct response would be scored as wrong). This highlights a massive "noise floor" that can make small improvements look like significant breakthroughs.
  • Generic is Often Better: In some cases, a simple "neutral note" (e.g., "Handle each request using the functions as needed") outperformed complex, learned lessons.

Key Results

The quantitative findings suggest that while failure traces help, their impact is limited:

  • Decision Accuracy: On isolated retail decisions, trace-based lessons beat the no-evidence search by +4.0 percentage points. While statistically significant, this was below the 5-point threshold the authors pre-registered as "worthwhile."
  • Conversation Success: In full retail conversations, the learned lessons showed no demonstrated improvement over the standard baseline.
  • BFCL Performance: No trace-written lessons passed the rigorous "Gate" for the tool-use benchmark. The one lesson that did pass (from the no-evidence search) failed to replicate its gains on held-out data.
Figure 1: (a) Decision-level (“Stage 1”) accuracy on 2,997 sealed 𝜏2 retail cases: model only (B0), standard harness (B1), no-evidence search (“T+ search control”), trace lessons (“T+ learned library”; T+ is the teacher writer) and a person-written reference. Markers are per-task values for the address, variant and sequence tasks. (b) Wrong writes (write calls that are not exactly the correct one) for the same arms. (c) Official 𝜏2 retail conversations (“Stage 2”; 114 tasks × 4 trials): pass^𝑘for the standard harness and the trace lessons. pass^𝑘is the chance that all 𝑘trials of a task succeed, so it cannot increase as 𝑘grows (unlike pass@𝑘, where any one success counts). The learned arm uses a different lesson library in each trial while the standard harness uses one prompt, so for 𝑘> 1 its pass^𝑘also reflects variation across libraries; the pass^4 gap is exploratory.

The headline results are summarized in the table below:

Table 1: Headline registered results. Estimates are in percentage points: contrasts are first arm minus second, gate rows a lesson’s estimated gain; intervals are 95% confidence intervals except those marked CrI (90% credible intervals). The BFCL held-out primary is standard harness −lesson, so there a positive value favours the standard harness. Trace lessons were written from the agent’s own episodes; the no-evidence search used the same writer and procedure without them.

Limitations and Future Work

The authors are careful to note several limitations:

  • Scope: The study focused on a single 9B model and one family of teacher models.
  • Rule Discovery vs. Phrasing: On the retail benchmark, the teacher already knew the target rules. The experiment measured how well the teacher could phrase those rules for the agent, not whether it could discover new rules from scratch.
  • Snapshot Decisions: Decision-level tests are "snapshots" that don't allow the agent to recover from errors, which might not reflect real-world agentic behavior.

The paper concludes with a call for more rigorous evaluation in the field: researchers should always report a matched "no-evidence" search control and perform blind audits of their scorers to ensure they aren't just measuring noise.

Summary updated

Abstract

We evaluate whether an AI agent's harness, the notes around its model, can learn from the agent's failure traces beyond a matched search that never sees them. It can, but less than it looks. We tested an open 9-billion-parameter model with fixed weights; a larger teacher model wrote its notes. On generated customer-service decisions, where both note writers were told the target rules, notes based on the agent's mistakes improved accuracy by 4.0 percentage points over notes from the same process with the mistakes withheld. That process alone, trying and keeping better-scoring notes, reached about 60% of the total improvement. In full customer-service conversations the same notes showed no demonstrated improvement. Separately, a strict screen of single lessons admitted two customer-service lessons, still awaiting tests on new data, and one tool-use lesson, written without seeing the agent's mistakes, which showed no demonstrated gain on new tool-use tasks. A neutral note with no procedure scored higher than the learned lessons (exploratory). A blind audit judged 146 of 240 tool-use test items defective.
BibTeX
@misc{mitra2026agent,
  author = {Arkid Mitra},
  title = {{Agent Self-Improvement with Frozen Weights: The Added Value and Limits of Failure Traces}},
  year = {2026},
  eprint = {2611.11342},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11342},
}
Figure