TL;DR
As environments evolve, the "skills" (reusable experience) stored by LLM agents can become stale, leading to significant performance drops or increased execution costs. This paper introduces selective maintenance, a framework that automatically detects environment-skill conflicts and repairs them, restoring agent efficiency and accuracy without requiring a full, expensive rebuild of the skill library.
What Problem Does This Paper Address?
Modern AI agents increasingly rely on "skills"—retained experience from past tasks—to solve new problems efficiently. By reusing successful procedures or data, agents can skip expensive reasoning steps. However, software environments are not static: APIs are renamed, form fields change, and configurations shift.
When a skill’s internal state no longer matches the environment, it becomes stale. The authors identify a critical gap: stale skills don't just fail; they can cause "negative transfer," where an agent performing with a stale skill actually does worse (or costs more) than an agent with no skill at all. Current systems often lack a principled way to detect and fix this "silent" degradation without human intervention or starting from scratch.
Methods
The authors propose a four-stage Selective Skill Maintenance pipeline designed to keep agent skills fresh:
- Profile Healthy Use: When a skill is first successfully executed in a "healthy" environment ($W_0$), the system records an execution profile. This includes "checks" (e.g., file existence, API responses) and links them to specific parts of the skill state.
- Replay Profiled Checks: Before reusing a skill in a new environment ($W_1$), the system replays these checks. Crucially, this step uses no LLM calls, making it extremely cheap.
- Verify & Localize: If differences are detected, an LLM "verifier" determines if the change actually conflicts with the skill. If it does, the verifier pinpoints exactly which instruction or code block is affected.
- Maintain & Validate: The system first attempts a selective repair of the localized conflict. If repair fails or the conflict is too broad, it triggers a re-induction (rebuilding the skill from scratch). Finally, the update is validated on a small set of tasks to ensure it actually restores performance before being committed.
Datasets
The researchers evaluated their approach across four diverse benchmarks, each with controlled environment transitions:
- TerminalBench-2 (TB-2): Command-line tasks where library versions or configurations change.
- IRS Forms: Tax-form filling tasks using historical revisions (e.g., 2018 to 2024) where fields are renamed or renumbered.
- AppWorld: API-based tasks where endpoints are renamed.
- ScienceWorld: Text-based science simulations where target objects are relocated.
Key Insights and Ideas
- The "Strong Model Trap": Stronger models (like Claude 3 Opus) are often able to detect and compensate for stale skills at runtime through extra reasoning. However, this "recovery" is expensive, often costing more in tokens and turns than if the agent had no skill at all.
- Negative Transfer is Real: In some cases, using a stale skill reduced task performance by over 10 percentage points compared to a baseline with no skills.
- Operational Specificity Matters: The paper compares "procedural" skills (which store how to find info) with "cache-heavy" skills (which store the info itself). Cache-heavy skills are faster when things are stable but break much harder when the environment shifts.
- Maintenance Pays for Itself: While maintenance has a one-time token cost, the researchers found this cost is typically "repaid" within 21 to 42 subsequent task executions due to the restored efficiency.
Key Results
The impact of selective maintenance was substantial across all benchmarks:
- Accuracy Recovery: On IRS tax forms, maintenance improved field accuracy by up to 67 percentage points compared to using stale state.
- Efficiency Gains: In ScienceWorld, maintenance reduced the number of environment steps required by up to 37%.
- Cost-Benefit: Selective maintenance consistently outperformed "No Maintenance" (reusing stale state) and was more efficient than "Retiring" (deleting the skill and starting over).
Limitations and Future Work
- Controlled Transitions: The study focused on "clean" environment shifts where one specific thing changed. Real-world environments might change many things simultaneously over long periods.
- Model Family: The evaluation was limited to the Claude 3 family (Opus and Haiku). Results might vary with other model architectures.
- Adapter Requirements: Each new type of environment (e.g., a new OS or a new web framework) requires a small "adapter" to define what its specific "checks" look like.
- Future Direction: The authors suggest exploring "automatic profile refresh," where the system continuously updates its ground-truth observations during normal operation rather than waiting for a maintenance trigger.
Summary updated