Before the Next Rollout: Diagnosing Skill Utility Shifts from Policy Updates

Yifan Wang, Zhenyi Gai, Yiwen Zhao, Mingxuan Jiang, Keyang Chen, Zeping Li, Hongfeng Chai, Yixin CaoarXiv 2026 · arXiv

TL;DR

As LLM agents are fine-tuned via reinforcement learning, the "skills" (external prompt-based guidance) they rely on can lose their effectiveness, even if the skill text remains unchanged. This paper introduces a method to diagnose these "utility shifts" using only existing training data and forward passes, allowing for targeted skill revision that improves agent performance while reducing token costs by over 75%.

What Problem Does This Paper Address?

LLM agents often use a library of reusable skills—structured procedures or tips—to handle complex tasks like web browsing or logical reasoning. However, these skills don't exist in a vacuum; their usefulness is tightly coupled with the agent's underlying "policy" (the LLM itself).

When we update an agent's policy (e.g., through Reinforcement Learning), a skill that was once a helpful crutch might become redundant or even confusing to the "smarter" updated model. This is known as a utility shift. Traditionally, identifying which skills need fixing requires running expensive new simulations (rollouts) to see where the agent fails. Furthermore, when an agent fails a task, it’s often unclear if the fault lies with a poorly written skill or the agent's own reasoning. The authors seek a way to diagnose these shifts "before the next rollout," using signals already present in the training process.

Figure 1: Policy updates alter skill marginal utility. Distributions before and after policy updates, with skill content fixed. Darker shades indicate higher concentra- tions of skills at the corresponding utility values.

Methods

The core of the paper is a diagnostic framework that reuses pre-update training rollouts to score skills without needing new environment interactions.

  1. Fixed-State Policy Readouts: The researchers take a specific decision point (context) from the training data. They compare how the old policy and the new policy react to that context under two conditions: with the skill guidance and with a "control" (where the skill is removed).
  2. Skill-Control Interaction Shift: By subtracting the "control" change from the "skill-conditioned" change, they isolate how the policy update specifically altered the agent's relationship with that particular skill.
  3. Reward-Directed Diagnosis: Simply knowing a behavior changed isn't enough; we need to know if it changed for the better. The authors project these interaction shifts onto a "reward direction" derived from training advantages (signals of which actions led to success). This produces a reward-directed readout—a score that predicts if a skill's utility is declining.
  4. Readout-Guided Revision: These scores prioritize which skills are sent to an LLM-based "editor" for revision. Instead of a blanket update, the editor focuses only on the skills most likely to be dragging down performance.
Figure 3: Overview of our method. With the skill library fixed within each update window, we reuse pre-update rollouts to compare action preferences under skill and matched-control conditions before and after the policy update. Combining these fixed-state comparisons with batch advantages yields reward-directed readouts that prioritize skills for revision without additional diagnostic rollouts.

Datasets

The authors evaluated their method across six diverse benchmarks:

  • ALFWorld: Embodied household tasks (e.g., "put a clean sponge in the cabinet").
  • WebShop: Constraint-aware online shopping.
  • HotpotQA & Natural Questions (NQ): Multi-hop and single-hop fact retrieval.
  • SpreadsheetBench: Code generation for spreadsheet manipulation.
  • LogicBench: Complex logical reasoning.

Key Insights and Ideas

  • Policy Updates are "Canaries": You don't need to see an agent fail in the wild to know a skill is becoming obsolete. The subtle shifts in how the LLM prefers certain tokens during an update are highly predictive of future performance drops.
  • Decoupling is Essential: By using a matched control condition (removing the skill), the method successfully separates the agent's general improvement from the specific marginal contribution of the skill guidance.
  • Efficiency via Prioritization: The "aha moment" here is that skill evolution is often a bottleneck due to the cost of LLM-based editing. By using readouts to identify the "squeaky wheels," the authors achieve better results with a fraction of the computational budget.

Key Results

  • High Diagnostic Accuracy: The reward-directed readouts showed a strong Spearman correlation (up to $\rho = 0.778$ on WebShop) with actual measured declines in skill utility.
  • Superior Performance: The method outperformed the strongest skill-evolution baselines by 6.07% on ALFWorld and 3.82% on LogicBench.
  • Massive Cost Savings: On the ALFWorld benchmark, the approach reduced the number of tokens consumed by the LLM editor by 76.7% (from 14.94M to 3.47M tokens) compared to methods that analyze all failures indiscriminately.
Figure 2: Diagnostic value and downstream benefits of policy-update readouts. (a) Association between reward-directed readout percentiles and measured utility declines across three benchmarks. Positive −∆M indicates declining skill utility; ρ denotes Spearman correlation. (b) ALFWorld unseen success rate versus policy token consumption under full-failure and readout-guided revision. Bubble size indicates cumulative editor token usage.

Limitations and Future Work

The authors acknowledge several constraints:

  • Data Dependency: The method relies on having access to training advantages and policy checkpoints, which might not be available in all black-box RL settings.
  • Sparse Signals: If a skill is rarely invoked during training, the readout may be less informative due to a lack of data points.
  • Fixed Routing: The current experiments use a fixed "router" to select skills. The authors suggest that extending this to evolving routers—where the agent learns which skill to pick while also refining the skills themselves—is a promising next step.

Summary updated

Abstract

Skill-evolution methods typically analyze execution trajectories to guide agent skill refinement. However, policy updates can change the behavioral effects and marginal utility of an unchanged skill. Attribution based on earlier trajectories may therefore no longer reflect the skill's contribution under the updated policy. Moreover, trajectory failures can arise from multiple factors, making individual skill contributions difficult to isolate. We evaluate whether policy-update signals contain predictive information about changes in skill marginal utility. At fixed contexts from pre-update training rollouts, we measure how policy updates change the contrast in token preferences between skill and control conditions. Combining these interaction shifts with advantage estimates yields reward-directed readouts for diagnosing utility changes. The readouts reuse existing training rollouts and require only forward passes for single-step action scoring, without additional post-update rollout evaluation. Our experiments show that policy updates can alter skill marginal utility and that policy-update readouts carry diagnostic information for identifying skills with declining utility. Integrating readout-guided prioritization into skill evolution improves downstream performance while reducing policy and editor token consumption.
BibTeX
@misc{wang2026before,
  author = {Yifan Wang and Zhenyi Gai and Yiwen Zhao and Mingxuan Jiang and Keyang Chen and Zeping Li and Hongfeng Chai and Yixin Cao},
  title = {{Before the Next Rollout: Diagnosing Skill Utility Shifts from Policy Updates}},
  year = {2026},
  eprint = {2610.39358},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39358},
}
Figure