TL;DR
CheckerBench is a new, executable benchmark designed to evaluate whether AI coding agents can synthesize reusable static-analysis checkers from scratch. Unlike previous benchmarks that focus on one-off bug fixes, this task requires long-horizon reasoning to navigate complex analyzer APIs and iteratively refine tools that catch entire classes of vulnerabilities. The results are a wake-up call: even the best current agents fail more than half the time, with a mean success rate of only 32.30%.
What Problem Does This Paper Address?
In the world of software security, a "patch" is a one-off fix for a specific bug. While useful, it doesn't prevent similar bugs from appearing elsewhere. For that, we need static-analysis checkers—automated tools that scan codebases for specific defect patterns (like a buffer overflow or a SQL injection).
Currently, writing these checkers is a high-effort task reserved for human experts who understand both the vulnerability and the arcane internals of static analysis frameworks (like Clang Static Analyzer or CodeQL). While LLM-based coding agents have become adept at generating patches, we don't know if they can graduate to building the tools that prevent bugs.
Existing benchmarks suffer from several "evaluation gaps," as illustrated in Figure 2:
- Not checker-centric: They focus on individual fixes rather than reusable tools.
- Limited interaction: They often use predefined, fixed stages rather than allowing agents to explore, fail, and debug iteratively.
- Fragmented signals: They don't measure critical factors like false positives or whether a checker actually localizes the bug correctly.
Methods
The authors introduce two major components: CheckerBench (the dataset) and CheckerLab (the evaluation framework).
The Synthesis Pipeline
The researchers developed a six-stage pipeline to convert real-world vulnerabilities (CVEs) into executable agent tasks. As shown in Figure 3, this involves recovering vulnerable/fixed code pairs, reconstructing the build environment, and providing the agent with a "checker scaffold"—a starting point for their code.
CheckerLab: The "Judge"
When an agent submits a checker, CheckerLab puts it through a rigorous, multi-gate evaluation:
- Build: Does the checker even compile?
- Contrast: Does it report a bug in the vulnerable version but not in the fixed version?
- Localization: Does it point to the actual lines of code modified by the patch?
- False Positive Control: Does it stay quiet when run against a large "mainline" codebase, or does it spam the user with false alarms?
Datasets
CheckerBench consists of 300 tasks derived from 297 unique CVEs. It is remarkably diverse, covering:
- 5 Language Ecosystems: C/C++, Java, Python, JavaScript, and Go.
- 167 Repositories: Including heavyweights like the Linux kernel and FFmpeg.
- 85 CWEs: Covering a wide range of weakness types, from memory corruption to cross-site scripting.
The distribution of these tasks is shown in Figure 4, highlighting that while C/C++ (using the Clang Static Analyzer) makes up the majority, the benchmark provides a broad look at the state of the art across the industry.
Key Insights and Ideas
- The "Harness" Matters: The performance of an LLM isn't just about the model itself; it's about the "scaffold" or harness it uses to interact with the environment. The same model can perform drastically differently depending on whether it uses a shell-only interface or a tool-assisted one.
- Patching $\neq$ Checking: An agent might be great at fixing a bug (patching) but terrible at writing a rule to catch it (checking). The "Pass@1" success rate for checker synthesis is significantly lower than what these models typically achieve on patching benchmarks like SWE-bench.
- C/C++ is the Final Boss: Agents struggled significantly more with C/C++ tasks using the Clang Static Analyzer (CSA) than with other languages using CodeQL. This suggests that the complexity of the analyzer's API is a major bottleneck for current AI.
- Skills are Essential: In ablation studies, removing the "skill library" (reusable strategies for interpreting patches and designing checkers) caused performance to plummet from nearly 40% to just 8%. Agents need more than just raw coding ability; they need domain-specific "know-how."
Key Results
The researchers evaluated 7 leading models (including GPT-4o, Claude 3.5, and DeepSeek-V4) across 3 different agent harnesses.
- Overall Performance: The mean Pass@1 across all configurations was 32.30%.
- The Leader: The best-performing combination was Claude Opus 4.8 using the OpenCode harness, reaching 45.33%.
- The Gap: There is a massive gap between a checker that simply "builds" (~78%) and one that actually passes all the semantic and false-positive gates (~32%).
Figure 1 shows the performance ladder, illustrating that while some models are clearly superior, no current agent is close to "solving" the benchmark.
Limitations and Future Work
The authors acknowledge several constraints:
- Time Budget: Agents were limited to 50 minutes. While Figure 8 shows diminishing returns after 40 minutes, longer horizons might yield different results.
- Analyzer Scope: The benchmark is currently tied to CSA and CodeQL. Expanding to other tools (like Semgrep or specialized kernel checkers) would increase its utility.
- Data Contamination: While the authors audited for prior exposure to CVEs and found that "de-identifying" tasks only caused a small drop in performance, the risk of models having seen the training data remains a perennial challenge for all LLM benchmarks.
Future work will likely focus on improving agentic "planning"—helping models bridge the gap between a high-level analysis plan and a bug-free implementation of complex static-analysis logic.
Summary updated