Evolve Where It Matters: Coverage-Guided Self-Evolution of Agent Skills

Boyin Tan, Jiadong Zhang, Xiaosong Ma, Chun Jason Xue, Xiaowei Huang, Youcheng SunarXiv 2026 · arXiv

TL;DR

SKILLPLUMBER is a new framework that allows LLM agents to self-evolve by pinpointing exactly which instructions in a "skill" led to failure. By using instruction-level coverage—identifying which steps were followed, ignored, or never reached—it distinguishes between a bad instruction and a failure to follow a good one, outperforming existing self-evolution methods by up to 14 percentage points.

What Problem Does This Paper Address?

LLM agents often rely on "skills," which are essentially sets of natural-language instructions (e.g., "Read the CSV, then convert currency to USD"). However, these skills aren't perfect. Sometimes the instructions are wrong, and sometimes the agent simply ignores them.

Existing self-evolution methods try to fix these skills by looking at a failed run as a whole. If the agent fails, the system looks at the entire execution trace and tries to guess what to change. The authors argue this is like trying to debug a program without knowing which lines of code actually executed. Without "coverage" data, an agent might rewrite a perfectly good instruction that it simply failed to follow, or it might keep repeating a confusing instruction that the agent will ignore again.

Figure 1: Run-level feedback and instruction-level coverage of the same failed run. (a) The outcome and trajectory show that the run failed, but tie none of its steps to an instruction of the skill. (b) Coverage maps the run onto every instruction of SKILL.md. The agent followed the currency and rounding instructions, never met a situation calling for the refund instruction, and did not follow the USD instruction at step 34, which the revision below targets.

Methods

The authors introduce SKILLPLUMBER, a framework that treats skill evolution like program debugging. It introduces three key components:

  1. Instruction-Level Coverage: An "outcome-blind" judge labels every instruction in a skill as follow, not-follow, or not covered based on the execution trace.
  2. Persistent RCA Agent: A Root Cause Analysis (RCA) agent uses these coverage labels to ask targeted questions. For a not-follow label, it asks: "Would following this have helped, and why was it skipped?" For a follow label in a failed run, it asks: "Is this instruction itself wrong or too vague?"
  3. The Maintainer: This component distills successful debugging strategies across different tasks into a shared library. Later tasks can then benefit from the "investigative experience" gained in earlier ones.
Figure 2: Overview of SKILLPLUMBER. A persistent RCA agent uses coverage, execution traces, and permitted verifier feedback to diagnose a failure and revise the skill. It records its hypothesis and expected effect; a fresh execution provides evidence for the next revision decision. A maintainer shares investigative strategies across tasks. Coverage supplies instruction-use evidence to this self-evolution process.

Datasets

The researchers evaluated SKILLPLUMBER across three diverse benchmarks using three different LLM "executors" (GPT-5.5, Claude Opus 4.8, and DeepSeek-V4-Pro):

  • SkillsBench: 87 tasks spanning professional domains like science and finance.
  • SkillLearnBench: 50 real-world tasks grouped into families.
  • SWE-Skills-Bench-Hard: A 70-task "hard" subset of software engineering challenges in real repositories.

Key Insights and Ideas

  • Coverage is a Repair Signal, Not Just a Metric: While coverage is usually used to measure test adequacy, here it serves as a direct guide for where to edit.
  • The "Why" Matters: Distinguishing between a "wrong instruction" and an "unfollowed instruction" is the "aha moment" of the paper. If an agent ignores a correct instruction, the fix is to make it more prominent or easier to follow, not to change the logic.
  • Persistence Prevents Regression: Because the RCA agent maintains a persistent workspace for each task, it can remember previous hypotheses and retract edits that didn't work, preventing the agent from "wandering" into worse skill versions.
  • Cross-Task Strategy: The "Maintainer" allows the system to learn that certain types of errors (like rounding issues or specific API quirks) are common, creating a meta-knowledge base for debugging.

Key Results

SKILLPLUMBER significantly outperformed baselines like SkillRevise and CoEvoSkills:

  • Higher Success Rates: It solved 64.7% to 70.5% of tasks across benchmarks, a gain of 8.7 to 14.0 percentage points over the strongest baseline.
  • Faster Recovery: As shown in the repair dynamics, SKILLPLUMBER continues to find fixes in later rounds where other methods plateau.
  • Ablation Impact: Removing the instruction-level coverage signal reduced the number of recovered failures by 20.8%, proving that the fine-grained feedback is the primary driver of success.
  • Efficiency: While it uses more tokens per revision (due to the agentic setup), it recovers more tasks, making the "cost per recovered failure" comparable to simpler methods.
Figure 3: Initial failures repaired by each revision round with GPT-5.5 and Codex. A round of CoEvoSkills is one check against the benchmark verifier.

Limitations and Future Work

  • LLM Judge Reliability: The system relies on an LLM to act as the "coverage judge." If the judge mislabels an instruction, the RCA agent may be led astray.
  • Instruction-Only Focus: Currently, SKILLPLUMBER only evolves the natural language instructions. The authors note that many skills include bundled scripts or code, and extending coverage to those scripts is a logical next step.
  • Token Usage: The multi-agent approach (RCA, Judge, Maintainer) is token-intensive. Future work could look at optimizing these sessions or using smaller, specialized models for the judging and maintenance roles.

Summary updated

Abstract

Agent skills package procedural knowledge as natural-language instructions, and a skill helps only when its instructions are correct and properly followed by the agent. Self-evolving agents revise their own skills from the runs they make, but existing methods judge each run as a whole, by its outcome or by where it went wrong, and cannot tell whether a failure comes from a wrong instruction or from one the agent did not follow. Instruction-level coverage was introduced to test skills and labels each instruction of a run as follow, not-follow, or not covered, which separates these causes. We leverage it for skill self-evolution with SkillPlumber, a framework in which an agent evolves its own skills where coverage shows they fall short. A persistent root cause analysis agent turns each label into a revision question and checks each revision against the next run, while a maintainer distills investigations across tasks into a shared strategy library. Across three benchmarks and three executors, SkillPlumber solves 64.7% to 70.5% of the tasks, 25.6 to 30.4 percentage points higher than the initial skill and 8.7 to 14.0 percentage points higher than the strongest baseline, and removing coverage reduces recovered failures by 20.8% on SkillsBench with GPT-5.5.
BibTeX
@misc{tan2026evolve,
  author = {Boyin Tan and Jiadong Zhang and Xiaosong Ma and Chun Jason Xue and Xiaowei Huang and Youcheng Sun},
  title = {{Evolve Where It Matters: Coverage-Guided Self-Evolution of Agent Skills}},
  year = {2026},
  eprint = {2611.25429},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25429},
}
Figure