TalentRL: Strategy-Guided Reinforcement Learning for Hierarchical Skill Routing

Huancheng Chen, Xiaodi Sun, Zhaoqiong Huang, Michelle Chang, Jingwen LuarXiv 2026 · arXiv

TL;DR

TalentRL is a new reinforcement learning framework that teaches LLM agents to think strategically by organizing their procedural memory into a hierarchy of high-level "talents" and fine-grained "skills." By using matched-intervention RL to isolate the value of strategic choices from raw execution, it achieves state-of-the-art performance on complex reasoning tasks and interactive agent benchmarks like ALFWorld and WebShop.

What Problem Does This Paper Address?

Current Large Language Model (LLM) agents often use "SkillBanks"—libraries of reusable procedures—to solve tasks. However, most systems treat these skills as a flat list, forcing the agent to retrieve individual skills without a clear high-level plan. This leads to two major issues:

  1. Lack of Strategic Guidance: Agents might retrieve relevant skills but lack the overarching strategy (e.g., "decompose this complex question" vs. "compare these two entities") to use them effectively.
  2. The Credit Assignment Problem: If an agent fails a task, was it because it chose the wrong strategy, retrieved the wrong skills, or just made a typo during execution? Standard reinforcement learning, which relies on a single terminal reward, can't distinguish between these factors, making it difficult to train agents to be truly "talented" strategists.

Methods

The authors introduce TalentRL, which implements a hierarchical approach to procedural memory:

  • Hierarchical Structure: It separates Talents (coarse-grained strategies like "Bridge-Entity Resolution") from Skills (fine-grained executable procedures).
  • Talent-Aware SFT: The model is first initialized using Supervised Fine-Tuning (SFT) on demonstrations where "talents" are explicitly labeled based on their proven utility in successful rollouts.
  • Matched-Intervention RL: To solve the credit assignment problem, TalentRL uses a novel training signal. For a given task, it generates an "anchor" rollout (using the chosen talent and skills) and compares its reward against "control" rollouts where specific components are withheld (e.g., removing the skills or the talent guidance). These reward contrasts isolate the specific contribution of the routing decision.
  • Group-Relative Advantages: While routing is trained via contrasts, downstream execution is optimized using group-relative advantages (similar to the GRPO algorithm), which compares different execution attempts for the same task to find the best path.
Figure 1. Overview of TalentRL. (A) The policy model selects a talent that guides downstream execution and provides a soft prior for similarity-based skill retrieval. (B) Reward contrasts across matched intervention rollouts train talent routing, while group-relative advantages optimize downstream execution, both updating the shared policy model. (C) Talent-aware SFT initializes talent routing and downstream execution from outcome-guided demonstrations.

Datasets

The framework was evaluated across three diverse domains:

  • Search7: A suite of seven open-domain and multi-hop question-answering benchmarks, including Natural Questions (NQ), TriviaQA, HotpotQA, and Bamboogle.
  • ALFWorld: A synthetic environment for embodied agents to perform household tasks (e.g., "put a clean sponge in the cabinet").
  • WebShop: A benchmark for web navigation and online shopping tasks.

Key Insights and Ideas

  • Strategy as a Soft Prior: Instead of using talents as a hard filter, TalentRL uses them as a "soft prior" during retrieval. The chosen talent gives a score bonus to associated skills, allowing the agent to stay flexible if a highly similar but "unassociated" skill is actually the best fit.
  • Interventions are Better than Raw Rewards: By systematically "breaking" parts of the agent's context (the interventions), the researchers can measure exactly how much a specific strategy helped. This makes the training signal much cleaner than just looking at whether the agent got the final answer right.
  • Qualitative Alignment: The model naturally learns to map specific talents to task structures. For example, it learns to use "Bridge-Entity Resolution" for multi-hop questions that require finding an intermediate entity.

Key Results

TalentRL consistently outperformed strong baselines (like SkillRL and Skill1) across multiple LLM backbones (Qwen2.5-7B, Qwen3-8B, and Llama-3.1-8B):

  • Search7: TalentRL achieved the highest average success rate on all three backbones, with particularly strong gains in multi-hop reasoning (e.g., a +4.8% lead over the next best method on the difficult Bamboogle dataset).
  • ALFWorld: It reached a near-perfect 99.3% overall success rate, outperforming specialized memory-based agents.
  • WebShop: It achieved a score of 90.7, the highest among all compared methods.
Figure 2. Talent selection of the final TalentRL policies. (a) Search7 test sets, as % of questions per dataset. (b) ALFWorld by task type over the seen (140) and unseen (134) splits. (c) WebShop on 500 goals. Percentages below 10% are omitted from the bars.

Limitations and Future Work

While powerful, TalentRL currently relies on a predefined catalog of talents and skill associations. Although these are updated during the RL process, the initial "vocabulary" of strategies is human-defined.

The authors suggest that future work should focus on:

  1. Automated Talent Discovery: Moving beyond predefined catalogs to let the agent discover and name its own high-level strategies directly from experience.
  2. Scaling: Testing the framework on even larger models and more diverse, real-world agentic domains.
  3. Efficiency: Reducing the computational overhead of generating multiple matched-intervention rollouts during training.

Summary updated

Abstract

Natural-language skills enable large language model (LLM) agents to reuse procedures across tasks, yet most systems retrieve individual skills from a flat SkillBank without explicitly modeling the high-level strategy that governs their selection and use. We introduce TalentRL, a strategy-guided reinforcement learning framework for hierarchical routing over procedural memory. Coarse-grained talents represent recurring strategies for skill use, while fine-grained skills provide executable procedures. For each task, the policy model selects a talent that guides execution and softly prioritizes its associated skills during similarity-based retrieval. However, since task outcomes depend jointly on talent selection, retrieved skill context, and downstream execution, terminal rewards alone cannot isolate the contribution of the routing decision. TalentRL therefore trains talent routing with reward contrasts between rollouts of the same task under matched interventions on the talent-conditioned context, and optimizes downstream execution with group-relative advantages. Across seven open-domain and multi-hop question-answering benchmarks, talent-aware supervised fine-tuning improves average success rate by 7.53%, 6.71%, and 9.21% over the corresponding zero-shot baselines for Qwen2.5-7B, Qwen3-8B, and Llama-3.1-8B, respectively. Reinforcement learning further improves these SFT checkpoints by 7.58% to 9.32%, and TalentRL achieves the highest average success rate among all compared methods on all three backbones. On the agentic benchmarks ALFWorld and WebShop, TalentRL achieves the highest ALFWorld success rate (99.3%) and the highest WebShop score (90.7) among all compared methods.
BibTeX
@misc{chen2026talentrl,
  author = {Huancheng Chen and Xiaodi Sun and Zhaoqiong Huang and Michelle Chang and Jingwen Lu},
  title = {{TalentRL: Strategy-Guided Reinforcement Learning for Hierarchical Skill Routing}},
  year = {2026},
  eprint = {2610.41143},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41143},
}
Figure