Experience Rot: Experienced Agents Fail Where Novices Succeed

Zhen Xu, Hang Yin, Zitao Zhang, Kevin Zhao, Xuefeng LiuarXiv 2026 · arXiv

TL;DR

As LLM agents gain experience through memory and conversation history, they often become overconfident and rigid. This paper introduces "Experience Rot," a phenomenon where experienced agents fail at tasks after silent tool updates because they blindly repeat past successful patterns instead of checking current documentation—often performing significantly worse than "novice" agents with no prior experience.

What Problem Does This Paper Address?

We are currently in an arms race to give AI agents better "memory"—conversation histories, RAG-based notes, and reusable code skills. The assumption is that more experience leads to better performance. However, software environments are dynamic; APIs change, field names are updated, and parameter meanings are reversed.

The authors tackle the problem of silent tool changes: updates that don't trigger an explicit error but cause the tool to behave differently (e.g., an API ignoring an old field name instead of throwing a 400 error). In these scenarios, "experienced" agents often suffer from a form of cognitive inertia, relying on their internal "script" of what worked before rather than observing the reality of the tool definition currently in front of them.

Figure 1: Experience rot. (a) Deployed interfaces in which one field has opposite meanings, and in which changes need not produce errors: Todoist’s API reference (Doist, 2026) and two Todoist MCP servers (A: greirson, 2025, B: mehularora8, 2025); Python, JavaScript and POSIX conventions (Python Software Foundation, 2026; MDN Web Docs, 2026; The Open Group, 2018); the MCP specification (Model Context Protocol, 2025); SilentProbe (Li et al., 2026b). (b) Two runs of DeepSeek-V4-Flash on the same final task with the same tool definition. Without experience, the agent uses the field the definition lists. With experience from version A, it repeats its earlier call; the service accepts the call, ignores the unknown field, and the agent reports success. (c) Success on the final task without experience, with valid experience, and with experience from before the change, when the documentation gives the range of a value but not its meaning (incomplete documentation; six models, four changes; Section 4). Error bars are one standard error.

Methods

The researchers created a controlled experimental framework using three synthetic services: a calendar, a task list, and a mail client. Each service had two versions (A and B) that differed by exactly one interface detail, such as:

  • Field renaming: attendees becomes guests.
  • Structural changes: end_time becomes duration_minutes.
  • Meaning reversal: Priority 1 changing from "highest" to "lowest."

They tested six modern LLMs (including DeepSeek-V4-Flash, Qwen3.8-Flash, and GLM-5.3-Flash) across three experimental "arms":

  1. No Experience (Novice): The agent starts the final task fresh.
  2. Valid Experience: The agent performs a similar task on the same version of the tool first.
  3. Out-of-date Experience: The agent performs a task on Version A, then the final task on Version B (the "rot" condition).

The study specifically looked at whether agents performed a lookup (searching documentation) or a read-back (verifying the state of the service after a write operation).

Datasets

The study utilized newly introduced benchmarks based on the Model Context Protocol (MCP), a standard for defining tool interfaces for LLMs. The researchers built three custom MCP-compliant services (Calendar, Task List, Mail) specifically designed to isolate the effects of interface changes. Unlike existing benchmarks that keep tools fixed, this dataset focuses on version-to-version transitions.

Key Insights and Ideas

  • Experience kills curiosity: Once an agent has a successful tool call in its memory, it almost entirely stops checking documentation. Lookup rates plummeted from nearly 100% in novices to less than 6% in experienced agents.
  • The "Silent Failure" Trap: Because many production APIs are designed to be "robust" by ignoring unknown parameters, agents receive a success message even when they've used an outdated field. The agent then confidently (and incorrectly) reports to the user that the task is done.
  • More experience makes it worse: The "rot" intensifies with volume. Agents with ten prior successful tasks were more likely to fail after a change than those with only one prior task.
  • Memory type doesn't matter: Experience rot appeared regardless of whether the experience was stored in conversation history, a memory file, retrieved RAG notes, or as a Python "skill" script.
  • Benchmarks might be lying to us: Many research papers use "hand-written" histories to evaluate agents. The authors found that these artificial histories often include "check" steps that real agents skip, significantly hiding the true cost of experience rot.

Key Results

The quantitative drop in performance is stark. In scenarios with incomplete documentation (where agents must rely on provided examples):

  • Novices succeeded in 63% of tasks.
  • Agents with valid experience reached 100% success.
  • Agents with out-of-date experience crashed to 23% success.
Figure 2: Success rates on the final task. Out-of-date experience leads to a massive performance drop compared to both valid experience and having no experience at all.

The most effective remedy was mandatory read-back (instructing the agent to verify the result of its action), which raised success to 91%. However, this comes at a high cost, more than doubling the number of model calls required. Simple prompts like "the service has been updated" were largely ineffective.

Figure 4: Success rates across different memory storage methods and amounts of experience. Note how performance degrades as the number of earlier tasks increases.

Limitations and Future Work

The authors acknowledge several limitations:

  • Synthetic Environment: The services used were small and changes were isolated to single details. Real-world API migrations are often much messier.
  • Minimal Agent Loop: The study used a basic agent loop. More complex frameworks with built-in "reflection" or "planning" steps might behave differently, though the authors' tests on "check the result" instructions suggest the core problem remains.
  • Silent vs. Explicit Errors: The study focused on silent failures. Future work could explore how agents adapt when tools provide explicit, helpful error messages versus silent ones.

The paper concludes that simply "storing what worked" is insufficient for long-term agent autonomy. Memory systems need active mechanisms to detect when a stored "truth" has become a lie.

Summary updated

Abstract

Agents are increasingly built to learn from experience: they keep conversation histories, memory files, retrieved notes and skills, and reuse what worked before. We show that this experience turns against them when their tools change silently. We test a range of models with tools defined as in the Model Context Protocol (MCP). After a silent change, experienced agents fail on most tasks, while agents new to the task mostly succeed. We call this failure Experience Rot. Experienced agents copy their earlier call instead of checking the current interface, even when the tool definition shows the change. Then they report success. When the documentation is incomplete, experience is what makes agents succeed. After a change, the same experience leaves them worse off than no experience at all. The failure appears in every form of memory we test and grows with the amount of experience. Simple remedies fall short: announcing the update barely helps, and pointing the agent to the documentation fails whenever the documentation does not spell out the answer.
BibTeX
@misc{xu2026experience,
  author = {Zhen Xu and Hang Yin and Zitao Zhang and Kevin Zhao and Xuefeng Liu},
  title = {{Experience Rot: Experienced Agents Fail Where Novices Succeed}},
  year = {2026},
  eprint = {2611.20922},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20922},
}
Figure