SoK: Assessing the Evolution of LLM Capability in Software Vulnerability Research

Amine Lbath, Manan Suri, Aurelien Delaitre, Vadim Okun, Massih-Reza Amini, Dinesh ManochaarXiv 2026 · arXiv

This "Systematization of Knowledge" (SoK) paper provides a rigorous, evidence-based look at how Large Language Model (LLM) agents are transforming software vulnerability research. By moving past vendor hype and scattered anecdotes, the authors establish a formal framework to measure the autonomy and correctness of AI agents in finding, validating, and patching security flaws.

TL;DR

The paper introduces a standardized autonomy scale (A0–A5) for security agents and finds that LLMs reached expert-level autonomy in discovering and patching real-world vulnerabilities by late 2025. While this has led to a doubling of public vulnerability disclosures (CVEs) in 2026, the actual time it takes to fix or exploit these bugs has not yet significantly accelerated, revealing a persistent human bottleneck in the remediation lifecycle.

What Problem Does This Paper Address?

The security community is currently debating a potential "Vulnpocalypse"—a surge in AI-discovered vulnerabilities that could overwhelm human maintainers. However, evidence for LLM capabilities is often fragmented across marketing blogs, academic papers, and bug disclosures, making it nearly impossible to compare different systems.

This paper addresses the lack of a common yardstick. It seeks to answer:

  1. How autonomous are these systems? (Can they work without a human "pilot"?)
  2. How correct are they? (Are they just generating "slop" or finding real bugs?)
  3. What is the real-world impact? (Is the vulnerability lifecycle actually changing?)

Methods

The authors developed two primary tools for their assessment:

  1. The Autonomy & Correctness Framework: Inspired by the SAE levels of driving automation, they defined six levels of autonomy (A0–A5). For example, A4 represents an agent completing a task on a target without any human intervention, while A5 describes an agent that can choose its own targets and recover from its own failures over long periods. Correctness is graded from C0 (mostly wrong) to C3 (expert precision on code previously audited by humans).
  2. The Agentic Evidence Pipeline: To build their "Capability Map," the authors used an LLM-based workflow to ingest and score over 53,000 sources, including academic papers, lab reports, and incident disclosures. This "meta-research" approach allowed them to systematize a massive volume of unstructured data.
Figure 2. The evidence pipeline. Lanes 2, 4 and 5 run Claude Code agents, each as a separate process. Lane 1 consists of scripts and a local LLM, lane 3 of a script, and lane 6 of the authors’ review.

Datasets

  • The Corpus: 53,057 unique sources (academic, vendor, and community reports).
  • Public Records: Over 300,000 CVE records (2018–2026) to track volume and severity.
  • Lifecycle Data: 1,417 advisories from the Zero Day Initiative (ZDI) and Chrome’s bug tracker to measure "time to fix," and CISA’s Known Exploited Vulnerabilities (KEV) catalog to measure "time to attack."
  • Benchmarks: Results from 12 security benchmarks (e.g., Cybench, CyberGym, PatchEval) were used as complementary evidence.

Key Insights and Ideas

  • The 8-Month Open-Weight Lag: Capabilities first demonstrated by "frontier" models (like Claude or GPT) typically diffuse to open-weight models (like DeepSeek or GLM) within about 8 months. This suggests that restricting access to top-tier models only provides a temporary defense.
  • Resurrecting "Ancient" Bugs: LLM agents are remarkably effective at finding severe vulnerabilities in "audited" code—bugs that have survived for a decade or more in widely used software. In 2026, reports of such "old" bugs increased 6 to 11 times compared to 2025.
  • The Remediation Gap: While AI is great at finding bugs, fixing them remains a bottleneck. Bulk reports sent to outside maintainers have a fix rate as low as 5%, compared to over 70% for bugs found by tools integrated directly into a developer's internal workflow.
  • Autonomy Milestones: By August 2025, agents reached A4·C2 (expert autonomy) for discovery and validation. By mid-2026, agents under evaluation were observed choosing their own targets and exploiting unknown vulnerabilities to move between systems (A5·C1).
Figure 1. Highest autonomy level of each capability by access tier on direct evidence, with its correctness category and the month when the pair was first shown on real software. Outlined marks are levels reached only on a benchmark (‡) or by inference (†), which the third line names. An asterisk marks values supported by the developer’s own report (E1).

Key Results

  • CVE Explosion: The number of disclosed vulnerabilities more than doubled in 2026. Monthly counts rose from ~4,000 in early 2026 to nearly 17,000 by October.
  • Severity Shift: The share of vulnerabilities rated "Critical" or "High" rose from 46% in 2025 to 56% in late 2026.
  • Static Timelines: Despite the surge in findings, the median time from report to fix remained steady between 96 and 119 days. Similarly, the time from disclosure to observed attack has not yet shifted noticeably in public records.
Trends in CVE records, severity, and lifecycle timing.

Limitations and Future Work

  • Remediation Measurement: Patching and patch validation are the least measured tasks in the literature. Most reports focus on discovery, leaving the "fix" side of the equation under-researched.
  • Lack of "Red vs. Blue" Testing: The authors found no public evaluations that directly test current mitigations against a dedicated LLM-based attacker.
  • The "Slop" Problem: While top-tier agents are precise, the barrier to entry is so low that maintainers are being flooded with low-quality, AI-generated bug reports, which may be contributing to the "Vulnpocalypse" anxiety.

The paper concludes that while the "Vulnpocalypse" is visible in disclosure volumes, the best defenses remain established practices: moving to memory-safe languages and automating internal discovery-to-patch pipelines to match the pace of AI-driven attackers.

Summary updated

Abstract

LLM-based agents are becoming increasingly capable at discovering, validating and patching vulnerabilities in real software, and vendors and researchers report that this is already changing how vulnerabilities are found and fixed. These claims, however, are scattered across papers, vendor reports and disclosures and are rarely comparable. We introduce a framework that rates these capabilities on an autonomy scale modeled on the levels of driving automation, and qualifies each claim by its correctness, its availability and the strength of its evidence. We use this framework to systematize the published evidence on LLM capability in vulnerability research. We find that agents discovered, validated, and patched vulnerabilities in real-world software without human intervention before the restricted frontier models of 2026, and that open-weight models subsequently attained comparable autonomy for discovery and validation. We then measure the consequences of evolving LLM security capabilities in a decade of public vulnerability records, advisories and exploitation data. In 2026 the number of disclosed vulnerabilities more than doubled, with an increasing share rated critical or high, and severe bugs that had survived a decade or more in widely used code were reported several times more often than before. Yet the time between the stages of a vulnerability's life, from report to fix and from disclosure to attack, has not shifted noticeably so far. We grade the mitigations proposed in the literature with an explicit rubric and draw guidance for each group of stakeholders from the best-supported ones.
BibTeX
@misc{lbath2026sok,
  author = {Amine Lbath and Manan Suri and Aurelien Delaitre and Vadim Okun and Massih-Reza Amini and Dinesh Manocha},
  title = {{SoK: Assessing the Evolution of LLM Capability in Software Vulnerability Research}},
  year = {2026},
  eprint = {2611.26601},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26601},
}
Figure