TL;DR
Software repositories are living organisms, yet most LLM coding agents rely on static skill libraries that quickly become obsolete. Skill Dreaming introduces a framework that treats Pull Request (PR) streams as a continuous source of "executable knowledge," allowing agents to evolve repository-specific skills alongside the codebase. By closing the loop between PR analysis and skill management, the framework boosts issue resolution on the SWE-bench Pro benchmark from 79% to 88%.
What Problem Does This Paper Address?
While LLM agents have become adept at localized code fixes, they often struggle with the "tribal knowledge" of large, evolving repositories. Coding conventions change, APIs are deprecated, and maintenance requirements (like updating documentation or changelogs) are frequently overlooked.
Existing skill-learning methods typically extract lessons from static datasets or execution traces in a "one-and-done" fashion. They lack a mechanism to handle version-specific evolution: a skill learned from a PR six months ago might be an anti-pattern today. Skill Dreaming addresses this gap by using the natural history of a repository—its PR stream—to continuously refine, specialize, and supersede skills.
Methods
Skill Dreaming operates as a three-stage, closed-loop framework:
- PR Record Construction: The system identifies relevant historical PRs and reconstructs them into "usable" records. This involves segmenting commits into coherent changes and classifying them (e.g., usable, repairable, or outdated) based on their compatibility with the current repository state.
- Skill Distillation: A fixed-weight LLM "writer" extracts implementation patterns and maintenance obligations from these records. Unlike simple code snippets, these "skills" include triggers (when to apply), procedures (how to implement), and validation steps (how to test).
- Skill Alignment & Evolution: This is the core innovation. Instead of just appending new skills to a list, the system performs alignment operations (see Figure 4). It determines if a new PR should Update an existing skill, Specialize it for a specific context, or Supersede it entirely if the old way of doing things is now deprecated.
The final "dreamed" skills are injected into the agent's prompt as repository-specific instructions, guiding it to not only write code but also maintain the "surface" of the repo (tests, docs, and configs).
Datasets
The authors evaluated Skill Dreaming on several high-bar benchmarks:
- SWE-bench Pro: A subset of real-world GitHub issues from complex repos like Ansible and qutebrowser.
- SWE-bench Verified: A curated set of 12 repositories used to ensure the reliability of resolved issues.
- Cross-Repository Set: Six diverse domains including CLI tools (Click, Black), web libraries (HTTPX), and data science tools (Arrow).
Key Insights and Ideas
- Alignment over Accumulation: Simply adding more skills to an agent's context leads to noise and conflict. The paper demonstrates that explicitly managing the relationships between skills (like supersession) is vital for performance.
- Maintenance as a First-Class Citizen: The authors introduce the Maintenance Surface F1 metric. It measures whether an agent remembers to update the "boring" but essential parts of a repo—like changelogs and examples—which are often captured in historical PRs but ignored by standard RAG approaches.
- PRs as Ground Truth: PRs are superior to raw commits for learning because they contain the "why" (descriptions) and the "critique" (review comments), providing a richer signal for what constitutes a "good" change in a specific codebase.
Key Results
Skill Dreaming consistently outperforms base agents and previous state-of-the-art skill-learning methods across multiple backbones (Claude Opus and Sonnet).
- Resolution Accuracy: On SWE-bench Pro, it improved the resolved rate from 79% to 88% (Figure 5).
- Reliability: It achieved a perfect 1.00 Safe Rate on SWE-bench Verified, meaning every generated PR satisfied basic safety and execution constraints.
- Continuous Improvement: As shown in Figure 2, the agent's performance generally trends upward as it processes more historical PRs, proving that the skill library successfully adapts to the repository's evolution.
Limitations and Future Work
Despite the gains, the paper notes a few hurdles:
- Computational Overhead: Skill Dreaming is more token-intensive and takes slightly longer per task than simpler agents due to the distillation and alignment phases.
- Localization Granularity: While the system is excellent at module and file-level localization, pinpointing the exact function-level changes remains a challenge (Table 6).
- Dependency on History: The quality of "dreamed" skills is inherently tied to the quality of the repository's PR history. "Messy" repos with poorly documented PRs may yield less effective skill libraries.
Future work may explore more efficient ways to manage these libraries or integrate human-in-the-loop feedback to further refine the "dreaming" process.
Summary updated