This "Systematization of Knowledge" (SoK) paper provides a rigorous, evidence-based look at how Large Language Model (LLM) agents are transforming software vulnerability research. By moving past vendor hype and scattered anecdotes, the authors establish a formal framework to measure the autonomy and correctness of AI agents in finding, validating, and patching security flaws.
TL;DR
The paper introduces a standardized autonomy scale (A0–A5) for security agents and finds that LLMs reached expert-level autonomy in discovering and patching real-world vulnerabilities by late 2025. While this has led to a doubling of public vulnerability disclosures (CVEs) in 2026, the actual time it takes to fix or exploit these bugs has not yet significantly accelerated, revealing a persistent human bottleneck in the remediation lifecycle.
What Problem Does This Paper Address?
The security community is currently debating a potential "Vulnpocalypse"—a surge in AI-discovered vulnerabilities that could overwhelm human maintainers. However, evidence for LLM capabilities is often fragmented across marketing blogs, academic papers, and bug disclosures, making it nearly impossible to compare different systems.
This paper addresses the lack of a common yardstick. It seeks to answer:
- How autonomous are these systems? (Can they work without a human "pilot"?)
- How correct are they? (Are they just generating "slop" or finding real bugs?)
- What is the real-world impact? (Is the vulnerability lifecycle actually changing?)
Methods
The authors developed two primary tools for their assessment:
- The Autonomy & Correctness Framework: Inspired by the SAE levels of driving automation, they defined six levels of autonomy (A0–A5). For example, A4 represents an agent completing a task on a target without any human intervention, while A5 describes an agent that can choose its own targets and recover from its own failures over long periods. Correctness is graded from C0 (mostly wrong) to C3 (expert precision on code previously audited by humans).
- The Agentic Evidence Pipeline: To build their "Capability Map," the authors used an LLM-based workflow to ingest and score over 53,000 sources, including academic papers, lab reports, and incident disclosures. This "meta-research" approach allowed them to systematize a massive volume of unstructured data.
Datasets
- The Corpus: 53,057 unique sources (academic, vendor, and community reports).
- Public Records: Over 300,000 CVE records (2018–2026) to track volume and severity.
- Lifecycle Data: 1,417 advisories from the Zero Day Initiative (ZDI) and Chrome’s bug tracker to measure "time to fix," and CISA’s Known Exploited Vulnerabilities (KEV) catalog to measure "time to attack."
- Benchmarks: Results from 12 security benchmarks (e.g., Cybench, CyberGym, PatchEval) were used as complementary evidence.
Key Insights and Ideas
- The 8-Month Open-Weight Lag: Capabilities first demonstrated by "frontier" models (like Claude or GPT) typically diffuse to open-weight models (like DeepSeek or GLM) within about 8 months. This suggests that restricting access to top-tier models only provides a temporary defense.
- Resurrecting "Ancient" Bugs: LLM agents are remarkably effective at finding severe vulnerabilities in "audited" code—bugs that have survived for a decade or more in widely used software. In 2026, reports of such "old" bugs increased 6 to 11 times compared to 2025.
- The Remediation Gap: While AI is great at finding bugs, fixing them remains a bottleneck. Bulk reports sent to outside maintainers have a fix rate as low as 5%, compared to over 70% for bugs found by tools integrated directly into a developer's internal workflow.
- Autonomy Milestones: By August 2025, agents reached A4·C2 (expert autonomy) for discovery and validation. By mid-2026, agents under evaluation were observed choosing their own targets and exploiting unknown vulnerabilities to move between systems (A5·C1).
Key Results
- CVE Explosion: The number of disclosed vulnerabilities more than doubled in 2026. Monthly counts rose from ~4,000 in early 2026 to nearly 17,000 by October.
- Severity Shift: The share of vulnerabilities rated "Critical" or "High" rose from 46% in 2025 to 56% in late 2026.
- Static Timelines: Despite the surge in findings, the median time from report to fix remained steady between 96 and 119 days. Similarly, the time from disclosure to observed attack has not yet shifted noticeably in public records.
Limitations and Future Work
- Remediation Measurement: Patching and patch validation are the least measured tasks in the literature. Most reports focus on discovery, leaving the "fix" side of the equation under-researched.
- Lack of "Red vs. Blue" Testing: The authors found no public evaluations that directly test current mitigations against a dedicated LLM-based attacker.
- The "Slop" Problem: While top-tier agents are precise, the barrier to entry is so low that maintainers are being flooded with low-quality, AI-generated bug reports, which may be contributing to the "Vulnpocalypse" anxiety.
The paper concludes that while the "Vulnpocalypse" is visible in disclosure volumes, the best defenses remain established practices: moving to memory-safe languages and automating internal discovery-to-patch pipelines to match the pace of AI-driven attackers.
Summary updated