PenForge: On-the-Fly Expert Agent Construction for Automated Penetration Testing

Huihui Huang, Jieke Shi, Yun Ma, Ting Zhang, Yikun Li, Chengran Yang, Eng Lieh Ouh, Lwin Khin Shar, David Lo

TL;DR

PenForge is a new framework for automated penetration testing that abandons the "one-size-fits-all" agent approach in favor of on-the-fly expert agent construction. By performing automated reconnaissance first and then dynamically building specialized agents tailored to the specific target, it achieved a 30% success rate on the challenging CVE-Bench—tripling the performance of previous state-of-the-art methods.

What Problem Does This Paper Address?

Penetration testing (pentesting) is the "ethical hacking" process used to find security holes before the bad guys do. Currently, it’s a manual, labor-intensive process that requires elite expertise. Naturally, researchers want to automate this using Large Language Models (LLMs).

However, existing LLM-based pentesting tools fall into two traps:

  1. The Generalist Trap: A single agent tries to do everything but gets overwhelmed by the complexity of real-world web apps.
  2. The Rigid Specialist Trap: Frameworks use pre-defined "expert" agents (e.g., one for SQL injection, one for XSS). These struggle in "zero-day" scenarios because they lack the specific context of the target application and can't adapt their strategies based on what they find during discovery.

The authors tackle the "zero-day" challenge: how can an AI successfully exploit a target when it is given nothing but a URL and a list of possible attack types?

Methods

The core innovation of PenForge is the Meta-Planner, a high-level orchestrator that operates in two distinct phases:

Phase I: Target Reconnaissance

Instead of attacking blindly, the Meta-Planner acts like a digital scout. It uses a toolset to map the target:

  • EndpointScanner: Uses feroxbuster to find hidden files and API endpoints.
  • WebPageReader: Scrapes HTML to understand the user interface.
  • KnowledgeRetriever: Queries the Perplexity API to get the latest external security knowledge about the technologies it identifies.

Phase II: Sequential Attack Attempts

Once the scout returns, the Meta-Planner doesn't just pick a tool; it constructs a specialist. It ranks the most likely vulnerabilities and then prompts an LLM to generate a custom AutoGPT execution script. This script includes a role summary, best practices, and—crucially—the specific context gathered in Phase I (like a specific login URL found during the scan).

If the first "expert" fails, the Meta-Planner learns from the logs, adjusts its ranking, and instantiates a new expert for the next most likely attack type.

Datasets

The researchers evaluated PenForge using CVE-Bench, a rigorous benchmark containing 40 real-world web vulnerabilities (CVEs) with high-risk CVSS scores (9.0+).

They focused on the zero-day setting, which is the most realistic and difficult mode. In this mode, the agent is not told which vulnerability exists or where it is; it must find and exploit it autonomously. They compared PenForge against three baselines: Cy-Agent (generic), AutoGPT (general purpose), and T-Agent (pre-defined specialists).

Key Insights and Ideas

  • Context-Awareness is the "Secret Sauce": The jump from 10% to 30% success suggests that how an agent is initialized matters more than just having a "specialist" label. By feeding the agent the results of the reconnaissance phase, PenForge gives the LLM a "map" that static agents lack.
  • The "Multi-Path" Nature of Vulnerabilities: The authors found that for some CVEs, PenForge succeeded using an attack type different from the one recorded in the benchmark. For example, a "Database Access" vulnerability might be successfully exploited via an "Unauthorized Administrator Login." This suggests that real-world systems often have overlapping attack surfaces.
  • The Bottleneck is Tool Usage, Not Logic: Qualitative analysis showed that when PenForge failed, it was often due to "tool misuse"—the LLM understood the strategy but fumbled the specific parameters or syntax of the hacking tools it was trying to use.

Key Results

  • 3x Performance Boost: PenForge achieved a 30.0% success rate (12/40) in the zero-day setting. For context, the previous best (T-Agent) hit only 10%, while generic agents like Cy-Agent struggled at 2.5%.
  • Success@1 vs. Success@5: Even on the first attempt (Success@1), PenForge had a 20.5% probability of success, more than double the strongest baseline.
  • Strongest Categories: The framework was particularly effective at exploiting "unauthorized administrator logins" and "outbound service" (SSRF) vulnerabilities, which accounted for 66.6% of its successful exploits. These are high-impact flaws that often expose explicit endpoints that PenForge’s reconnaissance phase is great at finding.

Limitations and Future Work

  • Improving Tool Proficiency: The authors plan to replace the simple knowledge retriever with a "pentesting-specific" version that provides environment-specific usage hints to reduce tool misuse.
  • Expanding the Benchmark: CVE-Bench currently maps one target to one CVE. In reality, one app might have five different holes. The authors want to evolve benchmarks to reflect this multi-vulnerability reality.
  • Human-in-the-Loop: To build trust with security teams, future versions will focus on "explainability"—making sure the agent can explain why it is taking a specific action so a human can review it before it executes.

Summary updated

Abstract

Penetration testing is essential for identifying vulnerabilities in web applications before real adversaries can exploit them. Recent work has explored automating this process with Large Language Model (LLM)-powered agents, but existing approaches either rely on a single generic agent that struggles in complex scenarios or narrowly specialized agents that cannot adapt to diverse vulnerability types. We therefore introduce PenForge, a framework that dynamically constructs expert agents during testing rather than relying on those prepared beforehand. By integrating automated reconnaissance of potential attack surfaces with agents instantiated on the fly for context-aware exploitation, PenForge achieves a 30.0% exploit success rate (12/40) on CVE-Bench in the particularly challenging zero-day setting, which is a 3 times improvement over the state-of-the-art. Our analysis also identifies three opportunities for future work: (1) supplying richer tool-usage knowledge to improve exploitation effectiveness; (2) extending benchmarks to include more vulnerabilities and attack types; and (3) fostering developer trust by incorporating explainable mechanisms and human review. As an emerging result with substantial potential impact, PenForge embodies the early-stage yet paradigm-shifting idea of on-the-fly agent construction, marking its promise as a step toward scalable and effective LLM-driven penetration testing.
BibTeX
@misc{huang2026penforge,
  author = {Huihui Huang and Jieke Shi and Yun Ma and Ting Zhang and Yikun Li and Chengran Yang and Eng Lieh Ouh and Lwin Khin Shar and David Lo},
  title = {{PenForge: On-the-Fly Expert Agent Construction for Automated Penetration Testing}},
  year = {2026},
  eprint = {2601.06910},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.06910},
}
Figure