TL;DR
As LLM agents gain experience through memory and conversation history, they often become overconfident and rigid. This paper introduces "Experience Rot," a phenomenon where experienced agents fail at tasks after silent tool updates because they blindly repeat past successful patterns instead of checking current documentation—often performing significantly worse than "novice" agents with no prior experience.
What Problem Does This Paper Address?
We are currently in an arms race to give AI agents better "memory"—conversation histories, RAG-based notes, and reusable code skills. The assumption is that more experience leads to better performance. However, software environments are dynamic; APIs change, field names are updated, and parameter meanings are reversed.
The authors tackle the problem of silent tool changes: updates that don't trigger an explicit error but cause the tool to behave differently (e.g., an API ignoring an old field name instead of throwing a 400 error). In these scenarios, "experienced" agents often suffer from a form of cognitive inertia, relying on their internal "script" of what worked before rather than observing the reality of the tool definition currently in front of them.
Methods
The researchers created a controlled experimental framework using three synthetic services: a calendar, a task list, and a mail client. Each service had two versions (A and B) that differed by exactly one interface detail, such as:
- Field renaming:
attendeesbecomesguests. - Structural changes:
end_timebecomesduration_minutes. - Meaning reversal: Priority
1changing from "highest" to "lowest."
They tested six modern LLMs (including DeepSeek-V4-Flash, Qwen3.8-Flash, and GLM-5.3-Flash) across three experimental "arms":
- No Experience (Novice): The agent starts the final task fresh.
- Valid Experience: The agent performs a similar task on the same version of the tool first.
- Out-of-date Experience: The agent performs a task on Version A, then the final task on Version B (the "rot" condition).
The study specifically looked at whether agents performed a lookup (searching documentation) or a read-back (verifying the state of the service after a write operation).
Datasets
The study utilized newly introduced benchmarks based on the Model Context Protocol (MCP), a standard for defining tool interfaces for LLMs. The researchers built three custom MCP-compliant services (Calendar, Task List, Mail) specifically designed to isolate the effects of interface changes. Unlike existing benchmarks that keep tools fixed, this dataset focuses on version-to-version transitions.
Key Insights and Ideas
- Experience kills curiosity: Once an agent has a successful tool call in its memory, it almost entirely stops checking documentation. Lookup rates plummeted from nearly 100% in novices to less than 6% in experienced agents.
- The "Silent Failure" Trap: Because many production APIs are designed to be "robust" by ignoring unknown parameters, agents receive a success message even when they've used an outdated field. The agent then confidently (and incorrectly) reports to the user that the task is done.
- More experience makes it worse: The "rot" intensifies with volume. Agents with ten prior successful tasks were more likely to fail after a change than those with only one prior task.
- Memory type doesn't matter: Experience rot appeared regardless of whether the experience was stored in conversation history, a memory file, retrieved RAG notes, or as a Python "skill" script.
- Benchmarks might be lying to us: Many research papers use "hand-written" histories to evaluate agents. The authors found that these artificial histories often include "check" steps that real agents skip, significantly hiding the true cost of experience rot.
Key Results
The quantitative drop in performance is stark. In scenarios with incomplete documentation (where agents must rely on provided examples):
- Novices succeeded in 63% of tasks.
- Agents with valid experience reached 100% success.
- Agents with out-of-date experience crashed to 23% success.
The most effective remedy was mandatory read-back (instructing the agent to verify the result of its action), which raised success to 91%. However, this comes at a high cost, more than doubling the number of model calls required. Simple prompts like "the service has been updated" were largely ineffective.
Limitations and Future Work
The authors acknowledge several limitations:
- Synthetic Environment: The services used were small and changes were isolated to single details. Real-world API migrations are often much messier.
- Minimal Agent Loop: The study used a basic agent loop. More complex frameworks with built-in "reflection" or "planning" steps might behave differently, though the authors' tests on "check the result" instructions suggest the core problem remains.
- Silent vs. Explicit Errors: The study focused on silent failures. Future work could explore how agents adapt when tools provide explicit, helpful error messages versus silent ones.
The paper concludes that simply "storing what worked" is insufficient for long-term agent autonomy. Memory systems need active mechanisms to detect when a stored "truth" has become a lie.
Summary updated