TL;DR
SkillBuffBench is a rigorous new benchmark designed to measure the real-world impact of "Agent Skills"—reusable packages of procedural knowledge—on LLM-based coding agents. Unlike previous evaluations that hand-pick the perfect skill for a task, SkillBuffBench tests agents in deployment-like conditions where they must automatically discover relevant skills from a library and manage persistent memory across multiple tasks.
What Problem Does This Paper Address?
As LLM agents move from simple chatbots to autonomous researchers, they are increasingly equipped with "Skills": directories containing instructions, scripts, and reference materials that the agent can load on demand. However, current evidence for the utility of these skills is flawed.
Most benchmarks use a "curated" approach where a human provides the exact skill needed for a specific task, starting from a blank slate. In the real world, two major factors complicate this:
- Discovery Cost: An agent must search through a library of mostly irrelevant skills to find the right one. Choosing the wrong skill can be worse than having none at all.
- Memory Confounding: Modern agent frameworks (harnesses) often persist information between sessions. If an agent "remembers" a successful strategy from a previous attempt, does the Skill still provide a benefit, or is it redundant?
Furthermore, the performance of an agent is often a "black box" interaction between the underlying Model (e.g., GPT-4) and the Harness (the software managing the tool loop and context window). SkillBuffBench aims to disentangle these effects.
Methods
The authors employ a paired factorial design to isolate the variables that actually drive agent success.
The Experimental Factors:
- 4 Harnesses: Ginkley, Codex, Claude Code, and OpenCode.
- 11 Models: A mix of proprietary SOTA models (Anthropic, Google, OpenAI) and leading open-weight models from Chinese providers (DeepSeek, Alibaba, etc.).
- 3 Skill Conditions:
- No Skill ($\emptyset$): Baseline with only default tools.
- Curated (S): The "upper bound" where the correct skill is pre-selected by a human.
- Automatic (A): The agent must discover and invoke the skill from a library.
- 2 Memory Modes: Memory off (M0) vs. Memory on (M1).
Evaluation Protocol:
Instead of using "LLM-as-a-judge," which can be biased or inconsistent, every task is marked by a deterministic verifier. These verifiers use domain-specific metrics (like root-mean-square residuals for data fitting or average precision for rankings) and map them to a continuous score between 0 and 1.
Datasets
The benchmark currently consists of 4 expert-contributed research tasks across two domains, with plans to expand on a rolling basis:
- Materials Science - SIMS Analysis: Extracting oxygen tracer diffusion coefficients from secondary-ion mass spectrometry data. This requires the agent to not just "fit a curve" but to judge if the data is sound.
- Materials Science - XRD Refinement: Identifying crystalline phases and refining unit-cell parameters from X-ray diffraction patterns using specialized software like GSAS-II.
- Biology - Regulatory Target Prioritization: Ranking candidate enhancer-gene pairs from CRISPR perturbation studies to propose the most informative follow-up experiments.
- Biology - Spheroid Imaging: Analyzing 3D fluorescence microscopy to measure the structure and cell distribution of tumor models.
Key Insights and Ideas
- The "Discovery Gap": By comparing the Curated (S) and Automatic (A) conditions, the benchmark quantifies exactly how much performance is lost when an agent has to find its own tools. This is the "practitioner's gap"—the difference between what a tool can do and what it actually does in the wild.
- Skills vs. Memory: The paper treats memory as a controlled factor. This allows researchers to see if Skills are just a "shortcut" for what an agent would eventually learn through memory, or if they provide unique procedural knowledge that memory cannot replicate.
- Harness-Model Crossing: By running every model under every harness, the benchmark reveals if a specific model is "lazy" or if its harness is simply failing to advertise available skills effectively.
- Trust over Truth: In scientific tasks, the "hard part" isn't just generating a number; it's knowing when the evidence is too thin to support a conclusion. The verifiers specifically reward agents that correctly identify insufficient data.
Key Results
Note: This version of the paper focuses on the benchmark design and protocol; the full evaluation of the 11-model pool is currently in progress.
However, the authors reference preliminary findings from a related study (SkillsBench) which showed that while curated skills can raise mean scores by 16.6 percentage points, the gains vary wildly by domain, and some tasks actually perform worse when skills are introduced. SkillBuffBench is designed to provide the granular data needed to explain why these variations occur.
Limitations and Future Work
- Task Volume: With only 4 tasks currently, the statistical power per domain is limited. The authors are actively soliciting more expert-contributed tasks.
- Rolling Basis: Because the "SOTA" in LLMs changes monthly, the model pool ($n=11$) is a moving target. The authors plan to update results continuously as new models (like GPT-6 or Claude 4) are released.
- Computational Cost: Running a full factorial design across 11 models, 4 harnesses, and multiple repeats results in thousands of runs (3,168 for the current pool), making this a high-resource benchmark to maintain.
Summary updated