SkillBuffBench: Measuring What Agent Skills Contribute across Harnesses, Models, Skill Access Modes and Memory

Gengyue Dong, Ye Wei, Zheng Zhu, Yidong Han, Jiale Miao, Xiaojun Jia, Guanghui LiarXiv 2026 · arXiv

TL;DR

SkillBuffBench is a rigorous new benchmark designed to measure the real-world impact of "Agent Skills"—reusable packages of procedural knowledge—on LLM-based coding agents. Unlike previous evaluations that hand-pick the perfect skill for a task, SkillBuffBench tests agents in deployment-like conditions where they must automatically discover relevant skills from a library and manage persistent memory across multiple tasks.

What Problem Does This Paper Address?

As LLM agents move from simple chatbots to autonomous researchers, they are increasingly equipped with "Skills": directories containing instructions, scripts, and reference materials that the agent can load on demand. However, current evidence for the utility of these skills is flawed.

Most benchmarks use a "curated" approach where a human provides the exact skill needed for a specific task, starting from a blank slate. In the real world, two major factors complicate this:

  1. Discovery Cost: An agent must search through a library of mostly irrelevant skills to find the right one. Choosing the wrong skill can be worse than having none at all.
  2. Memory Confounding: Modern agent frameworks (harnesses) often persist information between sessions. If an agent "remembers" a successful strategy from a previous attempt, does the Skill still provide a benefit, or is it redundant?

Furthermore, the performance of an agent is often a "black box" interaction between the underlying Model (e.g., GPT-4) and the Harness (the software managing the tool loop and context window). SkillBuffBench aims to disentangle these effects.

Methods

The authors employ a paired factorial design to isolate the variables that actually drive agent success.

Figure 1: SkillBuffBench at a glance. Expert-contributed research tasks, currently 4 in materials science and biology, are run under a paired factorial design that crosses four harnesses and multiple models, each model run under every harness, with three Skill conditions and two memory modes. A deterministic verifier marks every run; the effect of Skills and of memory is reported with paired bootstrap confidence intervals, next to reliability, efficiency and failure types.

The Experimental Factors:

  • 4 Harnesses: Ginkley, Codex, Claude Code, and OpenCode.
  • 11 Models: A mix of proprietary SOTA models (Anthropic, Google, OpenAI) and leading open-weight models from Chinese providers (DeepSeek, Alibaba, etc.).
  • 3 Skill Conditions:
    • No Skill ($\emptyset$): Baseline with only default tools.
  • Curated (S): The "upper bound" where the correct skill is pre-selected by a human.
  • Automatic (A): The agent must discover and invoke the skill from a library.
  • 2 Memory Modes: Memory off (M0) vs. Memory on (M1).

Evaluation Protocol:

Instead of using "LLM-as-a-judge," which can be biased or inconsistent, every task is marked by a deterministic verifier. These verifiers use domain-specific metrics (like root-mean-square residuals for data fitting or average precision for rankings) and map them to a continuous score between 0 and 1.

Figure 2: Evaluation protocol. (a) Each task is run under three Skill conditions that differ only in Skill access. The macro-average scores P∅, PS and PA give the curated gain ∆S and the automatic gain ∆A against the same baseline. (b) Every run starts from a reset environment, and the verifier sees only what the run leaves behind.

Datasets

The benchmark currently consists of 4 expert-contributed research tasks across two domains, with plans to expand on a rolling basis:

  1. Materials Science - SIMS Analysis: Extracting oxygen tracer diffusion coefficients from secondary-ion mass spectrometry data. This requires the agent to not just "fit a curve" but to judge if the data is sound.
  2. Materials Science - XRD Refinement: Identifying crystalline phases and refining unit-cell parameters from X-ray diffraction patterns using specialized software like GSAS-II.
  3. Biology - Regulatory Target Prioritization: Ranking candidate enhancer-gene pairs from CRISPR perturbation studies to propose the most informative follow-up experiments.
  4. Biology - Spheroid Imaging: Analyzing 3D fluorescence microscopy to measure the structure and cell distribution of tumor models.

Key Insights and Ideas

  • The "Discovery Gap": By comparing the Curated (S) and Automatic (A) conditions, the benchmark quantifies exactly how much performance is lost when an agent has to find its own tools. This is the "practitioner's gap"—the difference between what a tool can do and what it actually does in the wild.
  • Skills vs. Memory: The paper treats memory as a controlled factor. This allows researchers to see if Skills are just a "shortcut" for what an agent would eventually learn through memory, or if they provide unique procedural knowledge that memory cannot replicate.
  • Harness-Model Crossing: By running every model under every harness, the benchmark reveals if a specific model is "lazy" or if its harness is simply failing to advertise available skills effectively.
  • Trust over Truth: In scientific tasks, the "hard part" isn't just generating a number; it's knowing when the evidence is too thin to support a conclusion. The verifiers specifically reward agents that correctly identify insufficient data.

Key Results

Note: This version of the paper focuses on the benchmark design and protocol; the full evaluation of the 11-model pool is currently in progress.

However, the authors reference preliminary findings from a related study (SkillsBench) which showed that while curated skills can raise mean scores by 16.6 percentage points, the gains vary wildly by domain, and some tasks actually perform worse when skills are introduced. SkillBuffBench is designed to provide the granular data needed to explain why these variations occur.

Limitations and Future Work

  • Task Volume: With only 4 tasks currently, the statistical power per domain is limited. The authors are actively soliciting more expert-contributed tasks.
  • Rolling Basis: Because the "SOTA" in LLMs changes monthly, the model pool ($n=11$) is a moving target. The authors plan to update results continuously as new models (like GPT-6 or Claude 4) are released.
  • Computational Cost: Running a full factorial design across 11 models, 4 harnesses, and multiple repeats results in thousands of runs (3,168 for the current pool), making this a high-resource benchmark to maintain.

Summary updated

Abstract

Agent Skills, file-based packages of procedural knowledge that a coding agent can discover and load on demand, are now shipped by every major agent harness, yet the evidence for how much they help comes almost entirely from settings in which the right Skill is hand-picked for the task and the agent starts from a blank slate. Neither holds in deployment, where an agent chooses from whatever library happens to be installed and frequently carries memory from earlier work. We present SkillBuffBench, a benchmark and paired evaluation protocol that treats Skill access as a three-level factor (no Skill, a curated Skill, and automatic invocation from an installed library), crosses it with memory (off and on), with four agent harnesses and with multiple models (the state-of-the-art proprietary models from Anthropic, Google and OpenAI, and the state-of-the-art open-weight models from Chinese providers), every model being run under every harness, and evaluates the result on research tasks contributed by domain experts together with the Skills they use, currently in materials science and biology, with the suite extended on a rolling basis. Every run is marked by a deterministic verifier; uncertainty is reported with a paired, task-level bootstrap; and cost, agent rounds and wall-clock time are reported alongside the score rather than folded into it. This version of the paper describes the design of the benchmark, its evaluation protocol and its current tasks and Skills. The evaluation is in progress: results will be added once the runs are complete, and the task suite, results and analysis will continue to be updated on a rolling basis.
BibTeX
@misc{dong2026skillbuffbench,
  author = {Gengyue Dong and Ye Wei and Zheng Zhu and Yidong Han and Jiale Miao and Xiaojun Jia and Guanghui Li},
  title = {{SkillBuffBench: Measuring What Agent Skills Contribute across Harnesses, Models, Skill Access Modes and Memory}},
  year = {2026},
  eprint = {2611.22017},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22017},
}
Figure