TL;DR
This paper introduces SkillMaster, a novel framework that empowers LLM agents to autonomously learn, refine, and manage their own skill sets. Instead of relying on external teachers, SkillMaster enables agents to internalize skills through experience, leading to significant performance improvements and more adaptable AI agents.
What Problem Does This Paper Address?
Current LLM agents often rely on pre-defined skills or external "teachers" to guide their actions in complex tasks. This approach treats skills as external tools rather than inherent capabilities. The problem is that this limits the agent's ability to truly master tasks, adapt to new situations, and improve its own skill repertoire over time. Existing methods struggle to:
- Internalize skills: Agents can use skills, but they don't learn to create or refine them autonomously.
- Evaluate skill quality: It's hard to tell if a skill edit is genuinely helpful, as task success alone is a sparse and coarse signal.
- Jointly optimize: Balancing the learning of task execution and skill management is challenging and can lead to unstable training.
SkillMaster aims to solve these issues by making skill management an integral, learned part of the agent's policy.
Methods
SkillMaster introduces a three-pronged approach:
- Trajectory-Informed Skill Review: After each task episode, the agent reviews its own trajectory (actions, observations, outcomes). Based on this evidence, it uses tool calls to decide whether to propose a new skill, update an existing one, or keep the skill bank as is. This integrates skill management directly into the agent's learning loop.
- Counterfactual Skill Utility Reward: To objectively evaluate proposed skill changes, SkillMaster uses "probe tasks." These are related tasks that are run with both the original and the modified skill bank. The difference in performance (success and efficiency) on these probe tasks provides a direct, quantifiable signal of the skill edit's utility. This is a key innovation, moving beyond simple task success.
- DualAdv-GRPO (Dual-Advantage Generalized Proximal Policy Optimization): This is a specialized reinforcement learning algorithm. It decouples the advantage estimation for task-solving actions and skill-editing decisions. This separation stabilizes the joint training process, preventing interference between learning to act and learning to manage skills.
Datasets
SkillMaster was evaluated on two established benchmarks:
- ALFWorld: An embodied household manipulation benchmark with six task families (e.g., Pick and Place, Heat, Cool).
- WebShop: An online shopping benchmark requiring multi-step product search and purchase.
The paper does not introduce new benchmarks but uses existing ones to demonstrate the effectiveness of its approach.
Key Insights and Ideas
- Autonomous Skill Mastery is Achievable: The core idea is that LLM agents can learn to manage their own skill sets, moving beyond passive skill usage.
- Counterfactual Evaluation is Crucial: Simply succeeding or failing a task isn't enough to judge a skill. Evaluating skill edits on related "probe tasks" provides a much clearer signal of their true utility and impact on future performance.
- Internalization Over Externalization: SkillMaster aims to "internalize" skills into the agent's policy, reducing reliance on explicit skill retrieval at test time, which leads to more robust and adaptable agents.
- Decoupled Optimization is Key for Stability: Separating the learning signals for acting and skill management is essential for effectively training agents that perform both tasks simultaneously.
Key Results
SkillMaster demonstrated significant improvements over state-of-the-art baselines:
- ALFWorld: Achieved an 8.8% increase in overall success rate compared to the strongest baseline (SkillRL). It reached near-perfect performance (100%) on four out of six task families.
- WebShop: Showed a 9.3% improvement in overall success rate.
- Skill Internalization: Agents trained with SkillMaster performed nearly as well even when test-time skill retrieval was disabled, suggesting that procedural knowledge was effectively absorbed into the agent's policy.
- Robustness: SkillMaster maintained strong performance even with significantly weakened or entirely removed initial skill banks, highlighting its ability to learn and adapt skills from scratch.
Limitations and Future Work
- Probe Selection: The current probe selection relies on pre-defined task families or categories. For more open-ended environments, an adaptive similarity metric would be needed.
- Probe Evaluation Cost: Each skill mutation triggers additional probe rollouts, which can increase computational cost. Future work could explore methods like importance sampling or learned value functions to reduce this.
- Domain Scope: Experiments were conducted on structured benchmarks. Further validation in more open-ended domains like software engineering is needed.
- Future Directions: The authors suggest exploring skill deletion, multi-agent skill sharing, continual learning under distribution shifts, and extending the approach to other forms of agent self-improvement (e.g., prompt refinement, tool composition).
Summary updated