Skill Dreaming: Evolving Repository Skills through Pull Request Streams

Xinyuan Song, Hasibul Haque, Jay Somasundaram, Ryan Turner, Bowen Zhu, Liang ZhaoarXiv 2026 · arXiv

TL;DR

Software repositories are living organisms, yet most LLM coding agents rely on static skill libraries that quickly become obsolete. Skill Dreaming introduces a framework that treats Pull Request (PR) streams as a continuous source of "executable knowledge," allowing agents to evolve repository-specific skills alongside the codebase. By closing the loop between PR analysis and skill management, the framework boosts issue resolution on the SWE-bench Pro benchmark from 79% to 88%.

What Problem Does This Paper Address?

While LLM agents have become adept at localized code fixes, they often struggle with the "tribal knowledge" of large, evolving repositories. Coding conventions change, APIs are deprecated, and maintenance requirements (like updating documentation or changelogs) are frequently overlooked.

Existing skill-learning methods typically extract lessons from static datasets or execution traces in a "one-and-done" fashion. They lack a mechanism to handle version-specific evolution: a skill learned from a PR six months ago might be an anti-pattern today. Skill Dreaming addresses this gap by using the natural history of a repository—its PR stream—to continuously refine, specialize, and supersede skills.

Methods

Skill Dreaming operates as a three-stage, closed-loop framework:

  1. PR Record Construction: The system identifies relevant historical PRs and reconstructs them into "usable" records. This involves segmenting commits into coherent changes and classifying them (e.g., usable, repairable, or outdated) based on their compatibility with the current repository state.
  2. Skill Distillation: A fixed-weight LLM "writer" extracts implementation patterns and maintenance obligations from these records. Unlike simple code snippets, these "skills" include triggers (when to apply), procedures (how to implement), and validation steps (how to test).
  3. Skill Alignment & Evolution: This is the core innovation. Instead of just appending new skills to a list, the system performs alignment operations (see Figure 4). It determines if a new PR should Update an existing skill, Specialize it for a specific context, or Supersede it entirely if the old way of doing things is now deprecated.
Figure 1: Overview of Skill Dreaming: PR-driven skill evolution. Top: PR streams provide development records. Middle: Skills evolve through generation, reuse, revision, and replacement. Bottom: Evolving skills guide repository development and maintenance, generating feedback for further evolution.

The final "dreamed" skills are injected into the agent's prompt as repository-specific instructions, guiding it to not only write code but also maintain the "surface" of the repo (tests, docs, and configs).

Datasets

The authors evaluated Skill Dreaming on several high-bar benchmarks:

  • SWE-bench Pro: A subset of real-world GitHub issues from complex repos like Ansible and qutebrowser.
  • SWE-bench Verified: A curated set of 12 repositories used to ensure the reliability of resolved issues.
  • Cross-Repository Set: Six diverse domains including CLI tools (Click, Black), web libraries (HTTPX), and data science tools (Arrow).

Key Insights and Ideas

  • Alignment over Accumulation: Simply adding more skills to an agent's context leads to noise and conflict. The paper demonstrates that explicitly managing the relationships between skills (like supersession) is vital for performance.
  • Maintenance as a First-Class Citizen: The authors introduce the Maintenance Surface F1 metric. It measures whether an agent remembers to update the "boring" but essential parts of a repo—like changelogs and examples—which are often captured in historical PRs but ignored by standard RAG approaches.
  • PRs as Ground Truth: PRs are superior to raw commits for learning because they contain the "why" (descriptions) and the "critique" (review comments), providing a richer signal for what constitutes a "good" change in a specific codebase.

Key Results

Skill Dreaming consistently outperforms base agents and previous state-of-the-art skill-learning methods across multiple backbones (Claude Opus and Sonnet).

  • Resolution Accuracy: On SWE-bench Pro, it improved the resolved rate from 79% to 88% (Figure 5).
  • Reliability: It achieved a perfect 1.00 Safe Rate on SWE-bench Verified, meaning every generated PR satisfied basic safety and execution constraints.
  • Continuous Improvement: As shown in Figure 2, the agent's performance generally trends upward as it processes more historical PRs, proving that the skill library successfully adapts to the repository's evolution.
Figure 2: Continuous skill evolution through PR streams. TS, resolved rate, and Surface F1 of Skill Dreaming with Claude Opus 5 on SWE-bench Pro as new PRs arrive and repository-specific skills are incrementally updated.

Limitations and Future Work

Despite the gains, the paper notes a few hurdles:

  • Computational Overhead: Skill Dreaming is more token-intensive and takes slightly longer per task than simpler agents due to the distillation and alignment phases.
  • Localization Granularity: While the system is excellent at module and file-level localization, pinpointing the exact function-level changes remains a challenge (Table 6).
  • Dependency on History: The quality of "dreamed" skills is inherently tied to the quality of the repository's PR history. "Messy" repos with poorly documented PRs may yield less effective skill libraries.

Future work may explore more efficient ways to manage these libraries or integrate human-in-the-loop feedback to further refine the "dreaming" process.

Summary updated

Abstract

Large language model (LLM) agents increasingly rely on reusable skills for complex software engineering tasks, making skill libraries essential for repository development and maintenance. However, repository-specific skills must evolve with changing code structures, implementation conventions, and maintenance requirements. Pull requests (PRs) provide a natural source of such knowledge by recording how repositories evolve through implementation and review. Meanwhile, new PRs must follow repository-specific implementation patterns, modify related components consistently, and ensure that existing tests continue to pass while adding coverage for new functionality. This creates a mutually reinforcing cycle: PRs provide knowledge for updating skills, while these skills improve future PRs, maintain repositories, and support further skill evolution. To enable this co-evolution of PRs, repositories, and skills, we propose Skill Dreaming, a framework that learns from PR streams to guide continuous repository development and maintenance. Given a repository and its PR history, Skill Dreaming builds a repository-specific skill library by extracting executable knowledge from validated PRs and continuously refines the skills as new PR information becomes available. These skills guide coding agents not only in implementing requested features but also in maintaining associated tests, documentation, compatibility, and repository conventions. By integrating PR analysis, skill management, code execution, and continuous integration, Skill Dreaming provides an end-to-end workflow for applying learned skills to repository changes and incorporating subsequent PR outcomes into future skill updates.
BibTeX
@misc{song2026skill,
  author = {Xinyuan Song and Hasibul Haque and Jay Somasundaram and Ryan Turner and Bowen Zhu and Liang Zhao},
  title = {{Skill Dreaming: Evolving Repository Skills through Pull Request Streams}},
  year = {2026},
  eprint = {2610.36188},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36188},
}
Figure