TL;DR
SKILLPLUMBER is a new framework that allows LLM agents to self-evolve by pinpointing exactly which instructions in a "skill" led to failure. By using instruction-level coverage—identifying which steps were followed, ignored, or never reached—it distinguishes between a bad instruction and a failure to follow a good one, outperforming existing self-evolution methods by up to 14 percentage points.
What Problem Does This Paper Address?
LLM agents often rely on "skills," which are essentially sets of natural-language instructions (e.g., "Read the CSV, then convert currency to USD"). However, these skills aren't perfect. Sometimes the instructions are wrong, and sometimes the agent simply ignores them.
Existing self-evolution methods try to fix these skills by looking at a failed run as a whole. If the agent fails, the system looks at the entire execution trace and tries to guess what to change. The authors argue this is like trying to debug a program without knowing which lines of code actually executed. Without "coverage" data, an agent might rewrite a perfectly good instruction that it simply failed to follow, or it might keep repeating a confusing instruction that the agent will ignore again.
Methods
The authors introduce SKILLPLUMBER, a framework that treats skill evolution like program debugging. It introduces three key components:
- Instruction-Level Coverage: An "outcome-blind" judge labels every instruction in a skill as follow, not-follow, or not covered based on the execution trace.
- Persistent RCA Agent: A Root Cause Analysis (RCA) agent uses these coverage labels to ask targeted questions. For a
not-followlabel, it asks: "Would following this have helped, and why was it skipped?" For afollowlabel in a failed run, it asks: "Is this instruction itself wrong or too vague?" - The Maintainer: This component distills successful debugging strategies across different tasks into a shared library. Later tasks can then benefit from the "investigative experience" gained in earlier ones.
Datasets
The researchers evaluated SKILLPLUMBER across three diverse benchmarks using three different LLM "executors" (GPT-5.5, Claude Opus 4.8, and DeepSeek-V4-Pro):
- SkillsBench: 87 tasks spanning professional domains like science and finance.
- SkillLearnBench: 50 real-world tasks grouped into families.
- SWE-Skills-Bench-Hard: A 70-task "hard" subset of software engineering challenges in real repositories.
Key Insights and Ideas
- Coverage is a Repair Signal, Not Just a Metric: While coverage is usually used to measure test adequacy, here it serves as a direct guide for where to edit.
- The "Why" Matters: Distinguishing between a "wrong instruction" and an "unfollowed instruction" is the "aha moment" of the paper. If an agent ignores a correct instruction, the fix is to make it more prominent or easier to follow, not to change the logic.
- Persistence Prevents Regression: Because the RCA agent maintains a persistent workspace for each task, it can remember previous hypotheses and retract edits that didn't work, preventing the agent from "wandering" into worse skill versions.
- Cross-Task Strategy: The "Maintainer" allows the system to learn that certain types of errors (like rounding issues or specific API quirks) are common, creating a meta-knowledge base for debugging.
Key Results
SKILLPLUMBER significantly outperformed baselines like SkillRevise and CoEvoSkills:
- Higher Success Rates: It solved 64.7% to 70.5% of tasks across benchmarks, a gain of 8.7 to 14.0 percentage points over the strongest baseline.
- Faster Recovery: As shown in the repair dynamics, SKILLPLUMBER continues to find fixes in later rounds where other methods plateau.
- Ablation Impact: Removing the instruction-level coverage signal reduced the number of recovered failures by 20.8%, proving that the fine-grained feedback is the primary driver of success.
- Efficiency: While it uses more tokens per revision (due to the agentic setup), it recovers more tasks, making the "cost per recovered failure" comparable to simpler methods.
Limitations and Future Work
- LLM Judge Reliability: The system relies on an LLM to act as the "coverage judge." If the judge mislabels an instruction, the RCA agent may be led astray.
- Instruction-Only Focus: Currently, SKILLPLUMBER only evolves the natural language instructions. The authors note that many skills include bundled scripts or code, and extending coverage to those scripts is a logical next step.
- Token Usage: The multi-agent approach (RCA, Judge, Maintainer) is token-intensive. Future work could look at optimizing these sessions or using smaller, specialized models for the judging and maintenance roles.
Summary updated