TL;DR
PenForge is a new framework for automated penetration testing that abandons the "one-size-fits-all" agent approach in favor of on-the-fly expert agent construction. By performing automated reconnaissance first and then dynamically building specialized agents tailored to the specific target, it achieved a 30% success rate on the challenging CVE-Bench—tripling the performance of previous state-of-the-art methods.
What Problem Does This Paper Address?
Penetration testing (pentesting) is the "ethical hacking" process used to find security holes before the bad guys do. Currently, it’s a manual, labor-intensive process that requires elite expertise. Naturally, researchers want to automate this using Large Language Models (LLMs).
However, existing LLM-based pentesting tools fall into two traps:
- The Generalist Trap: A single agent tries to do everything but gets overwhelmed by the complexity of real-world web apps.
- The Rigid Specialist Trap: Frameworks use pre-defined "expert" agents (e.g., one for SQL injection, one for XSS). These struggle in "zero-day" scenarios because they lack the specific context of the target application and can't adapt their strategies based on what they find during discovery.
The authors tackle the "zero-day" challenge: how can an AI successfully exploit a target when it is given nothing but a URL and a list of possible attack types?
Methods
The core innovation of PenForge is the Meta-Planner, a high-level orchestrator that operates in two distinct phases:
Phase I: Target Reconnaissance
Instead of attacking blindly, the Meta-Planner acts like a digital scout. It uses a toolset to map the target:
- EndpointScanner: Uses
feroxbusterto find hidden files and API endpoints. - WebPageReader: Scrapes HTML to understand the user interface.
- KnowledgeRetriever: Queries the Perplexity API to get the latest external security knowledge about the technologies it identifies.
Phase II: Sequential Attack Attempts
Once the scout returns, the Meta-Planner doesn't just pick a tool; it constructs a specialist. It ranks the most likely vulnerabilities and then prompts an LLM to generate a custom AutoGPT execution script. This script includes a role summary, best practices, and—crucially—the specific context gathered in Phase I (like a specific login URL found during the scan).
If the first "expert" fails, the Meta-Planner learns from the logs, adjusts its ranking, and instantiates a new expert for the next most likely attack type.
Datasets
The researchers evaluated PenForge using CVE-Bench, a rigorous benchmark containing 40 real-world web vulnerabilities (CVEs) with high-risk CVSS scores (9.0+).
They focused on the zero-day setting, which is the most realistic and difficult mode. In this mode, the agent is not told which vulnerability exists or where it is; it must find and exploit it autonomously. They compared PenForge against three baselines: Cy-Agent (generic), AutoGPT (general purpose), and T-Agent (pre-defined specialists).
Key Insights and Ideas
- Context-Awareness is the "Secret Sauce": The jump from 10% to 30% success suggests that how an agent is initialized matters more than just having a "specialist" label. By feeding the agent the results of the reconnaissance phase, PenForge gives the LLM a "map" that static agents lack.
- The "Multi-Path" Nature of Vulnerabilities: The authors found that for some CVEs, PenForge succeeded using an attack type different from the one recorded in the benchmark. For example, a "Database Access" vulnerability might be successfully exploited via an "Unauthorized Administrator Login." This suggests that real-world systems often have overlapping attack surfaces.
- The Bottleneck is Tool Usage, Not Logic: Qualitative analysis showed that when PenForge failed, it was often due to "tool misuse"—the LLM understood the strategy but fumbled the specific parameters or syntax of the hacking tools it was trying to use.
Key Results
- 3x Performance Boost: PenForge achieved a 30.0% success rate (12/40) in the zero-day setting. For context, the previous best (T-Agent) hit only 10%, while generic agents like Cy-Agent struggled at 2.5%.
- Success@1 vs. Success@5: Even on the first attempt (Success@1), PenForge had a 20.5% probability of success, more than double the strongest baseline.
- Strongest Categories: The framework was particularly effective at exploiting "unauthorized administrator logins" and "outbound service" (SSRF) vulnerabilities, which accounted for 66.6% of its successful exploits. These are high-impact flaws that often expose explicit endpoints that PenForge’s reconnaissance phase is great at finding.
Limitations and Future Work
- Improving Tool Proficiency: The authors plan to replace the simple knowledge retriever with a "pentesting-specific" version that provides environment-specific usage hints to reduce tool misuse.
- Expanding the Benchmark: CVE-Bench currently maps one target to one CVE. In reality, one app might have five different holes. The authors want to evolve benchmarks to reflect this multi-vulnerability reality.
- Human-in-the-Loop: To build trust with security teams, future versions will focus on "explainability"—making sure the agent can explain why it is taking a specific action so a human can review it before it executes.
Summary updated