TL;DR
As LLM agents are fine-tuned via reinforcement learning, the "skills" (external prompt-based guidance) they rely on can lose their effectiveness, even if the skill text remains unchanged. This paper introduces a method to diagnose these "utility shifts" using only existing training data and forward passes, allowing for targeted skill revision that improves agent performance while reducing token costs by over 75%.
What Problem Does This Paper Address?
LLM agents often use a library of reusable skills—structured procedures or tips—to handle complex tasks like web browsing or logical reasoning. However, these skills don't exist in a vacuum; their usefulness is tightly coupled with the agent's underlying "policy" (the LLM itself).
When we update an agent's policy (e.g., through Reinforcement Learning), a skill that was once a helpful crutch might become redundant or even confusing to the "smarter" updated model. This is known as a utility shift. Traditionally, identifying which skills need fixing requires running expensive new simulations (rollouts) to see where the agent fails. Furthermore, when an agent fails a task, it’s often unclear if the fault lies with a poorly written skill or the agent's own reasoning. The authors seek a way to diagnose these shifts "before the next rollout," using signals already present in the training process.
Methods
The core of the paper is a diagnostic framework that reuses pre-update training rollouts to score skills without needing new environment interactions.
- Fixed-State Policy Readouts: The researchers take a specific decision point (context) from the training data. They compare how the old policy and the new policy react to that context under two conditions: with the skill guidance and with a "control" (where the skill is removed).
- Skill-Control Interaction Shift: By subtracting the "control" change from the "skill-conditioned" change, they isolate how the policy update specifically altered the agent's relationship with that particular skill.
- Reward-Directed Diagnosis: Simply knowing a behavior changed isn't enough; we need to know if it changed for the better. The authors project these interaction shifts onto a "reward direction" derived from training advantages (signals of which actions led to success). This produces a reward-directed readout—a score that predicts if a skill's utility is declining.
- Readout-Guided Revision: These scores prioritize which skills are sent to an LLM-based "editor" for revision. Instead of a blanket update, the editor focuses only on the skills most likely to be dragging down performance.
Datasets
The authors evaluated their method across six diverse benchmarks:
- ALFWorld: Embodied household tasks (e.g., "put a clean sponge in the cabinet").
- WebShop: Constraint-aware online shopping.
- HotpotQA & Natural Questions (NQ): Multi-hop and single-hop fact retrieval.
- SpreadsheetBench: Code generation for spreadsheet manipulation.
- LogicBench: Complex logical reasoning.
Key Insights and Ideas
- Policy Updates are "Canaries": You don't need to see an agent fail in the wild to know a skill is becoming obsolete. The subtle shifts in how the LLM prefers certain tokens during an update are highly predictive of future performance drops.
- Decoupling is Essential: By using a matched control condition (removing the skill), the method successfully separates the agent's general improvement from the specific marginal contribution of the skill guidance.
- Efficiency via Prioritization: The "aha moment" here is that skill evolution is often a bottleneck due to the cost of LLM-based editing. By using readouts to identify the "squeaky wheels," the authors achieve better results with a fraction of the computational budget.
Key Results
- High Diagnostic Accuracy: The reward-directed readouts showed a strong Spearman correlation (up to $\rho = 0.778$ on WebShop) with actual measured declines in skill utility.
- Superior Performance: The method outperformed the strongest skill-evolution baselines by 6.07% on ALFWorld and 3.82% on LogicBench.
- Massive Cost Savings: On the ALFWorld benchmark, the approach reduced the number of tokens consumed by the LLM editor by 76.7% (from 14.94M to 3.47M tokens) compared to methods that analyze all failures indiscriminately.
Limitations and Future Work
The authors acknowledge several constraints:
- Data Dependency: The method relies on having access to training advantages and policy checkpoints, which might not be available in all black-box RL settings.
- Sparse Signals: If a skill is rarely invoked during training, the readout may be less informative due to a lack of data points.
- Fixed Routing: The current experiments use a fixed "router" to select skills. The authors suggest that extending this to evolving routers—where the agent learns which skill to pick while also refining the skills themselves—is a promising next step.
Summary updated