SkillDuet: Co-Evolving Models and Skills through Response Feedback

Xingguang Ji, Ziqiang Dong, Yubo Zhang, Xi Leng, Yang Yang, Guanjun Jiang

TL;DR

SkillDuet is a novel framework for LLM agents that simultaneously updates model parameters and evolves a persistent library of textual "skills." By using model response feedback to select skills that most significantly shift the model's behavior, it ensures that external guidance remains relevant and beneficial even as the model's internal capabilities grow, leading to state-of-the-art performance on complex agentic benchmarks.

What Problem Does This Paper Address?

LLM agents typically improve through two distinct paths: internalization (updating model parameters via fine-tuning or reinforcement learning) and externalization (retaining experience as textual "skills" or prompts).

The challenge is that these two paths are rarely coordinated. A skill that provides excellent guidance early in training might become redundant once the model has "internalized" that specific capability into its parameters. Even worse, static skills can become detrimental if they conflict with newly learned behaviors. As shown in the authors' initial experiments (Figure 2), the performance boost from a fixed set of skills diminishes over time as the model improves.

The benefit of fixed skill guidance decreases as the model's internal parameters improve during training.

SkillDuet addresses this "moving target" problem by creating a feedback loop where the model and its skills co-evolve.

Methods

The SkillDuet framework (illustrated in Figure 3) couples two processes:

  1. Parameter Learning (The "Internal" Path): The agent interacts with an environment, guided by a set of "Global" skills. An external analyzer extracts "Local" skills from these interactions. These Local skills are used for On-Policy Distillation (OPD), providing token-level supervision to update the model's parameters at every step.
  2. Skill Evolution (The "External" Path): This is where SkillDuet introduces its most novel component: Strength-based selection. Instead of just keeping every successful skill, the framework ranks candidate Local skills by their "Strength"—the magnitude of the shift they induce in the model's current response probabilities.

The intuition is powerful: if a skill barely changes how the model responds, the model has likely already internalized that knowledge. A high "Strength" score indicates a skill that still has something new to teach the model. Periodically (every 10 steps), the highest-scoring skills are consolidated into the persistent Global skill set, which then guides the next round of interactions.

The SkillDuet framework coordinates parameter learning via OPD and skill evolution via response-guided selection.

Datasets

The authors evaluated SkillDuet using the Qwen3-4B model on two popular agentic benchmarks:

  • ALFWorld: A synthetic environment for household tasks (e.g., "put a clean knife on the table") that requires multi-step reasoning and interaction.
  • WebShop: A large-scale online shopping environment where agents must navigate a website to find and purchase specific items based on user requirements.

Key Insights and Ideas

  • Skill Utility is Dynamic: The value of a skill is not absolute; it's relative to the model's current state. This necessitates a mechanism for "forgetting" or de-prioritizing skills that the model has already mastered internally.
  • Response Shift as a Selection Signal: Measuring the "Strength" of a skill (how much it changes the model's mind) is a more effective filter for long-term retention than simply looking at task success or rewards. As Figure 4 shows, the Strength criterion significantly outperforms random selection or reward-weighted "Utility" scores.
  • Compatibility Matters: A skill set learned for one model state might actually hurt performance if applied to a different model state (or a different model entirely). This highlights the importance of the "co-evolution" aspect—the skills and parameters must grow together.
Comparison of different skill selection criteria. Strength achieves the highest peak validation success and area under the curve (AUC).

Key Results

SkillDuet achieved impressive gains over strong baselines:

  • ALFWorld: Reached a 98.57% success rate, outperforming the SEED-API baseline by 2.86 percentage points and achieving 100% success in five out of six task categories.
  • WebShop: Achieved a 92.97% success rate, a significant 5.47 percentage point improvement over the baseline.
  • Ablation Studies: The authors found that a 50% retention ratio (keeping the top half of skills by Strength) provided the best balance between retaining useful knowledge and avoiding redundancy (Figure 5).
The effect of different skill retention ratios on performance. Top50% was found to be the optimal default.

Limitations and Future Work

While SkillDuet represents a major step toward recursive self-improvement in agents, some limitations remain:

  • Computational Overhead: The framework relies on an external "analyzer" model (Qwen3.8-27B) to extract and synthesize skills, which adds cost and latency.
  • Heuristic Scoring: While "Strength" is effective, it's a heuristic that measures the magnitude of change but not necessarily the correctness of that change (though the authors argue this is mitigated by the fact that skills are extracted from successful trajectories).
  • Future Directions: The authors suggest exploring more efficient scoring methods that don't require full log-probability computations and investigating more sophisticated ways to resolve conflicts between overlapping or contradictory skills during the consolidation phase.

Summary updated

Abstract

During reinforcement learning, experience can be both internalized in LLM parameters and retained externally as reusable textual skills. Parameter updates alter the model's behavior and the interaction experience from which new skills are extracted. These skills, in turn, guide the model's subsequent interactions and parameter updates. As the model's capabilities evolve, previously beneficial skills may become redundant or detrimental. We present SkillDuet, a framework that coordinates parameter updates and skill evolution through model response feedback. Skills extracted from on-policy trajectories guide parameter updates through OPD at every step. A response-based score, Strength, ranks candidate skills by the magnitude of the shift they induce in the current model's responses, and the highest-scoring skills are periodically consolidated into a persistent global skill set (Global). The updated model and Global jointly guide subsequent interactions, generating experience for further parameter and skill updates. With Qwen3-4B, SkillDuet achieves success rates of 98.57% on ALFWorld and 92.97% on WebShop, exceeding the matched SEED-API baseline by 2.86 and 5.47 percentage points, respectively. Ablations show that response-based skill selection and Global consolidation improve task performance. Further experiments show that the benefits of retained skills depend on both their content and their compatibility with the model. SkillDuet provides a concrete mechanism for model-skill co-evolution and skill-level recursive self-improvement.
BibTeX
@misc{ji2026skillduet,
  author = {Xingguang Ji and Ziqiang Dong and Yubo Zhang and Xi Leng and Yang Yang and Guanjun Jiang},
  title = {{SkillDuet: Co-Evolving Models and Skills through Response Feedback}},
  year = {2026},
  eprint = {2611.12109},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12109},
}
Figure