AUSO: Action-Level Unified Skill Optimization from Internalization to Utilization

Huizu Lin, Chengkai Huang, Tianqi Gao, Tao Huang, Daijiao Liu, Tongxin Li, Xiaoyan Sun, Lina YaoarXiv 2026 · arXiv

TL;DR

AUSO (Action-level Unified Skill Optimization) is a progressive reinforcement learning framework that unifies how LLM agents learn ("internalize") and use ("utilize") procedural skills. By using Jensen-Shannon Divergence to measure skill relevance at the individual action level rather than the entire task level, AUSO dynamically rewards helpful skill usage and suppresses distractions. This fine-grained approach leads to massive performance leaps and superior out-of-distribution generalization across complex, long-horizon tasks.

What Problem Does This Paper Address?

To tackle complex, multi-step tasks—like online shopping or navigating an embodied environment—Large Language Model (LLM) agents rely on "skills" (reusable procedural knowledge, tool-use strategies, or heuristics). Current state-of-the-art hybrid methods try to balance learning these skills internally versus keeping them as external prompts. They typically use trajectory-level success rates to route tasks: hard tasks get external skill guidance, while easy tasks are used to internalize the skill into the model's weights.

However, this trajectory-level routing introduces two major flaws:

  1. Fuzzy Boundaries: As shown in Figure 1, task-level success rates are noisy. A tiny fluctuation can push a sample across an arbitrary threshold, completely changing its training objective.
  2. Coarse Granularity: Trajectory routing treats every action within a single attempt identically. In reality, a skill might be incredibly helpful for a pivotal decision but redundant or distracting for others.

Because environments only provide a single reward at the end of a trajectory, existing methods struggle with credit assignment—pinpointing exactly which actions actually benefited from the skill.

Fuzzy boundaries in trajectory-level routing

Methods

AUSO solves this by replacing discrete task routing with a continuous, progressive lifecycle built on a single reinforcement learning backbone: Group Relative Policy Optimization (GRPO).

Instead of toggling between learning and using skills, AUSO guides the agent through three progressive stages using an action-level counterfactual information operator based on Jensen-Shannon Divergence (JSD):

  1. Early-stage Teacher-guided Skill Internalization: When tasks are too difficult and rewards are sparse, a teacher model (conditioned on general skills) guides a skill-free student model. AUSO calculates the JSD between their action distributions, applying stronger corrective updates to actions where the student deviates most from the skilled teacher.
  2. Outcome-driven Autonomous Exploration: As the agent improves, teacher guidance is annealed. The agent explores independently using standard GRPO to solidify its autonomous problem-solving capabilities.
  3. Later-stage Action-level Skill Utilization: For a mature agent, AUSO evaluates each action under two contexts: with the skill and without the skill. The JSD between these two distributions measures "skill sensitivity."

To ensure stable updates, this sensitivity signal is modulated by a Global Uncertainty Gate. This gate peaks when a mix of successful and failed trajectories coexist, allowing the model to precisely reinforce actions that truly drive success while suppressing harmful or distracting skill applications.

Figure 2: Overview of Our AUSO (Action-Level Unified Skill Optimization from Internalization to Utilization): (1) General Skills Internalization: Internalize transferable general skills through action-level teacher–student guidance. (2) Skills Exploration: Explore and optimize diverse skills via GRPO. (3) Specific Skill Utilization: Evaluate the benefit of task-specific skills for each action to guide action-level skill utilization.

Datasets

The authors evaluated AUSO across three distinct long-horizon benchmarks:

  • ALFWorld: An embodied household task benchmark requiring text-based planning, navigation, and object manipulation.
  • WebShop: A simulated e-commerce environment where agents must search, compare, and purchase items based on user instructions.
  • SearchQA: A knowledge-intensive, search-based question-answering dataset.

To evaluate cross-domain generalization, the authors established strict category-based In-Distribution (ID) and Out-of-Distribution (OOD) splits for both ALFWorld and WebShop.

Key Insights and Ideas

  • Skills as a Spectrum, Not a Toggle: Rather than treating skill internalization and utilization as competing objectives, AUSO demonstrates that they can be unified under a single information-theoretic metric (JSD) across a continuous training lifecycle.
  • The Sweet Spot of Uncertainty: The Global Uncertainty Gate elegantly leverages the principle that binary outcomes are most informative when the empirical success rate is around 0.5. If an agent always succeeds or always fails, individual actions provide little contrast for credit assignment, and the gate naturally dials down the skill modulation.
  • Efficiency Through Compactness: Action-level optimization prevents the agent from over-relying on skills. This results in highly goal-directed trajectories, eliminating the redundant search steps and "over-answering" behaviors common in baseline agents.

Key Results

AUSO consistently outperformed competitive baselines (including Skill0.5, SkillRL, and standard GRPO) across all three benchmarks:

  • WebShop: AUSO achieved the highest average success rates, outperforming the previous best hybrid method (Skill0.5) by 9.3 points in-distribution and 10.6 points out-of-distribution.
  • ALFWorld: AUSO reached an ID average of 94.3% and an OOD average of 67.9% (surpassing Skill0.5 by 9.4 points on unseen tasks). It showed massive improvements on complex, multi-object tasks like Pick2.
  • SearchQA: AUSO achieved the highest overall average score of 47.5%, proving that action-level skill optimization transfers effectively to purely knowledge-intensive reasoning tasks.

Limitations and Future Work

While AUSO significantly improves sample efficiency and performance, calculating counterfactual distributions (evaluating actions both with and without skills) at every step introduces computational overhead during the training phase. Although the authors mitigated this using a coarse-grained top-$k$ token approximation, reducing the training-time token cost remains an open challenge.

Future work includes exploring more lightweight sensitivity estimators and extending this action-level framework to multi-agent collaboration settings.

Summary updated

Abstract

Skills play different roles as an agent's policy evolves: they should first provide learnable knowledge, then support capability formation, and finally be invoked only when they improve individual decisions. Existing methods rarely model this lifecycle. They either keep skills outside the model, fully internalize them, or select among internalization and utilization objectives through noisy task-level success rates. Such designs fragment training and assign uniform importance to actions within the same trajectory, even though skill guidance may help some decisions while distracting others. To solve these problems, we introduce AUSO (Action-level Unified Skill Optimization), which unifies skill learning and skill use through a progressive, action-aware optimization process. At the beginning of training, AUSO jointly learns from teacher guidance and environmental outcomes, enabling the policy to acquire foundational skills without losing task-oriented feedback. It subsequently emphasizes outcome-based policy optimization to consolidate autonomous problem-solving ability. As the policy matures, AUSO evaluates each sampled action under both skill-conditioned and skill-free contexts. The resulting action-level information signal is coupled with the trajectory outcome advantage, allowing beneficial skill-sensitive actions to receive stronger updates and harmful ones to be suppressed. Therefore, skills gradually transition from an external source of supervision into decision knowledge whose utilization is adapted to its action-level benefit, while reinforcement learning remains the shared backbone across all stages. Experiments on ALFWorld, WebShop, and SearchQA show that AUSO consistently improves agent performance and out-of-distribution generalization over competitive baselines.
BibTeX
@misc{lin2026auso,
  author = {Huizu Lin and Chengkai Huang and Tianqi Gao and Tao Huang and Daijiao Liu and Tongxin Li and Xiaoyan Sun and Lina Yao},
  title = {{AUSO: Action-Level Unified Skill Optimization from Internalization to Utilization}},
  year = {2026},
  eprint = {2608.21292},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.21292},
}
Figure