TL;DR
AUSO (Action-level Unified Skill Optimization) is a progressive reinforcement learning framework that unifies how LLM agents learn ("internalize") and use ("utilize") procedural skills. By using Jensen-Shannon Divergence to measure skill relevance at the individual action level rather than the entire task level, AUSO dynamically rewards helpful skill usage and suppresses distractions. This fine-grained approach leads to massive performance leaps and superior out-of-distribution generalization across complex, long-horizon tasks.
What Problem Does This Paper Address?
To tackle complex, multi-step tasks—like online shopping or navigating an embodied environment—Large Language Model (LLM) agents rely on "skills" (reusable procedural knowledge, tool-use strategies, or heuristics). Current state-of-the-art hybrid methods try to balance learning these skills internally versus keeping them as external prompts. They typically use trajectory-level success rates to route tasks: hard tasks get external skill guidance, while easy tasks are used to internalize the skill into the model's weights.
However, this trajectory-level routing introduces two major flaws:
- Fuzzy Boundaries: As shown in Figure 1, task-level success rates are noisy. A tiny fluctuation can push a sample across an arbitrary threshold, completely changing its training objective.
- Coarse Granularity: Trajectory routing treats every action within a single attempt identically. In reality, a skill might be incredibly helpful for a pivotal decision but redundant or distracting for others.
Because environments only provide a single reward at the end of a trajectory, existing methods struggle with credit assignment—pinpointing exactly which actions actually benefited from the skill.
Methods
AUSO solves this by replacing discrete task routing with a continuous, progressive lifecycle built on a single reinforcement learning backbone: Group Relative Policy Optimization (GRPO).
Instead of toggling between learning and using skills, AUSO guides the agent through three progressive stages using an action-level counterfactual information operator based on Jensen-Shannon Divergence (JSD):
- Early-stage Teacher-guided Skill Internalization: When tasks are too difficult and rewards are sparse, a teacher model (conditioned on general skills) guides a skill-free student model. AUSO calculates the JSD between their action distributions, applying stronger corrective updates to actions where the student deviates most from the skilled teacher.
- Outcome-driven Autonomous Exploration: As the agent improves, teacher guidance is annealed. The agent explores independently using standard GRPO to solidify its autonomous problem-solving capabilities.
- Later-stage Action-level Skill Utilization: For a mature agent, AUSO evaluates each action under two contexts: with the skill and without the skill. The JSD between these two distributions measures "skill sensitivity."
To ensure stable updates, this sensitivity signal is modulated by a Global Uncertainty Gate. This gate peaks when a mix of successful and failed trajectories coexist, allowing the model to precisely reinforce actions that truly drive success while suppressing harmful or distracting skill applications.
Datasets
The authors evaluated AUSO across three distinct long-horizon benchmarks:
- ALFWorld: An embodied household task benchmark requiring text-based planning, navigation, and object manipulation.
- WebShop: A simulated e-commerce environment where agents must search, compare, and purchase items based on user instructions.
- SearchQA: A knowledge-intensive, search-based question-answering dataset.
To evaluate cross-domain generalization, the authors established strict category-based In-Distribution (ID) and Out-of-Distribution (OOD) splits for both ALFWorld and WebShop.
Key Insights and Ideas
- Skills as a Spectrum, Not a Toggle: Rather than treating skill internalization and utilization as competing objectives, AUSO demonstrates that they can be unified under a single information-theoretic metric (JSD) across a continuous training lifecycle.
- The Sweet Spot of Uncertainty: The Global Uncertainty Gate elegantly leverages the principle that binary outcomes are most informative when the empirical success rate is around 0.5. If an agent always succeeds or always fails, individual actions provide little contrast for credit assignment, and the gate naturally dials down the skill modulation.
- Efficiency Through Compactness: Action-level optimization prevents the agent from over-relying on skills. This results in highly goal-directed trajectories, eliminating the redundant search steps and "over-answering" behaviors common in baseline agents.
Key Results
AUSO consistently outperformed competitive baselines (including Skill0.5, SkillRL, and standard GRPO) across all three benchmarks:
- WebShop: AUSO achieved the highest average success rates, outperforming the previous best hybrid method (Skill0.5) by 9.3 points in-distribution and 10.6 points out-of-distribution.
- ALFWorld: AUSO reached an ID average of 94.3% and an OOD average of 67.9% (surpassing Skill0.5 by 9.4 points on unseen tasks). It showed massive improvements on complex, multi-object tasks like Pick2.
- SearchQA: AUSO achieved the highest overall average score of 47.5%, proving that action-level skill optimization transfers effectively to purely knowledge-intensive reasoning tasks.
Limitations and Future Work
While AUSO significantly improves sample efficiency and performance, calculating counterfactual distributions (evaluating actions both with and without skills) at every step introduces computational overhead during the training phase. Although the authors mitigated this using a coarse-grained top-$k$ token approximation, reducing the training-time token cost remains an open challenge.
Future work includes exploring more lightweight sensitivity estimators and extending this action-level framework to multi-agent collaboration settings.
Summary updated