AutoPentester: An LLM Agent-based Framework for Automated Pentesting

Yasod Ginige, Akila Niroshan, Sajal Jain, Ranga Rodrigo2025 · DOI

TL;DR

This paper introduces AutoPentester, a novel LLM-agent framework that significantly automates penetration testing. By dynamically generating attack strategies and minimizing human intervention, AutoPentester outperforms existing LLM-based tools like PentestGPT in subtask completion, vulnerability coverage, and efficiency.

What Problem Does This Paper Address?

The cybersecurity landscape is facing an ever-increasing volume and sophistication of threats, leading to a surge in demand for penetration testing (pentesting) and vulnerability assessments. However, the cybersecurity industry faces a critical shortage of skilled professionals, making it difficult to meet this demand effectively. While AI, particularly Large Language Models (LLMs), has shown promise in automating aspects of pentesting, existing LLM-based tools like PentestGPT still require substantial human interaction and guidance. These tools often struggle with dynamic strategy generation, limited automation, and can only handle a restricted set of security tools, thus failing to fully automate the complex and iterative process of pentesting.

Methods

AutoPentester is a novel LLM-agent framework designed to mimic the human pentester's iterative approach. It comprises five key LLM-based agents:

  • Summarizer Agent: Processes and condenses lengthy tool outputs into a human-readable format for other agents.
  • Strategy Analyzer Agent: This is the core strategic planner. It updates a "Pentest Tree" (PTT) with findings from previous steps and uses Chain-of-Thought (CoT) reasoning to dynamically formulate the next attack strategy. This is a key differentiator from previous methods that relied on more static approaches.
  • Generator Agent: Generates executable commands for various cybersecurity tools. It leverages Retrieval-Augmented Generation (RAG) by querying a knowledge base of security resources to ensure accurate and contextually relevant command generation.
  • Agent Computer Interface (ACI): Acts as an intermediary, translating LLM-generated commands into executable scripts for common command-line interface (CLI) security tools and capturing their outputs.
  • Results Verifier Agent: Validates tool outputs and refines commands if necessary, adding flexibility and addressing potential issues like filtered ports.
  • Repetition Identifier Module: Prevents the framework from getting stuck in repetitive loops by detecting and flagging similar steps, thus improving efficiency.
  • Report Generator Agent: Compiles all gathered information into a structured .csv report, a standard industry practice.

Datasets

The paper evaluates AutoPentester using two types of target environments:

  • Hack The Box (HTB) Machines: A collection of 10 industry-grade pentesting training machines (6 easy, 4 intermediate) were used to assess AutoPentester's strategic pentesting abilities. Five of these machines were also used in the PentestGPT study for direct comparison.
  • Custom Virtual Machines (VMs): Four custom VMs were created, containing vulnerabilities from the OWASP Top 10. These were specifically designed to evaluate AutoPentester's performance in vulnerability and threat assessment, focusing on maximizing vulnerability identification and attack surface coverage. Metasploitable II was used as one of these VMs.

Key Insights and Ideas

  • Dynamic Strategy Generation: The Strategy Analyzer's ability to dynamically update its attack plan based on real-time tool outputs is a significant advancement over static or semi-manual approaches.
  • LLM as a Strategic Planner: Moving beyond simple command generation, AutoPentester leverages LLMs for higher-level strategic reasoning, mimicking human pentester thought processes.
  • RAG for Command Accuracy: The integration of RAG in the Generator agent significantly improves the accuracy and completeness of generated commands by grounding them in a relevant knowledge base.
  • Human-like Iteration: The framework's iterative nature, coupled with the Repetition Identifier, effectively prevents the common pitfall of LLMs getting stuck in repetitive, unproductive loops.
  • Bridging the Gap to Automation: AutoPentester demonstrates a substantial leap towards fully automated pentesting, drastically reducing the need for human intervention compared to prior LLM-based tools.

Key Results

  • Quantitative Performance:
    • Subtask Completion: AutoPentester achieved a 27.0% higher average subtask completion rate (59.92%) compared to PentestGPT (47.18%) on HTB machines.
  • Vulnerability Coverage: On custom VMs, AutoPentester achieved 98.14% vulnerability coverage on average, significantly outperforming PentestGPT's 70.37%.
  • Efficiency: AutoPentester required approximately 18.7% fewer steps than PentestGPT.
  • Loop Reduction: AutoPentester reduced repetitive loops by 85.7% (0.3 loops per machine on average) compared to PentestGPT (2.1 loops per machine).
  • Human Interaction: AutoPentester required significantly less human interaction, averaging 1.13 steps compared to PentestGPT's 15.36.
  • Incomplete Commands: AutoPentester generated only 0.1 incomplete commands per machine on average, a 97.7% reduction compared to PentestGPT.
  • Qualitative Findings:
    • User Study: Professional pentester volunteers rated AutoPentester higher (3.93/5) than PentestGPT (3.3/5), with participants finding AutoPentester's workflow structured, logical, and its remediation steps clear.
  • Report Quality: Participants found AutoPentester's reports to be more aligned with industry practices, offering better attack surface coverage and information revelation.
  • Automation Value: Professionals believed AutoPentester could save significant time in basic pentesting and red teaming tasks.

Limitations and Future Work

  • Limitations:
    • GUI Interaction: AutoPentester currently struggles with interacting with GUI-based interfaces, primarily relying on CLI tools like curl, which can lead to information loss.
  • Knowledge Base Scope: The RAG module's effectiveness is dependent on the knowledge base. If it misses specific exploits or requires external information (e.g., from GitHub repositories), AutoPentester may fail.
  • User Study Sample Size: While valuable, the user study's sample size could be larger for more robust conclusions.
  • Future Work:
    • GUI Interaction Module: Developing a module to interact with GUI-based interfaces is a key area for improvement.
  • LLM Fine-tuning for Strategy: Fine-tuning LLMs specifically for identifying complex strategies in pentesting tasks, potentially using Reinforcement Learning (RL) techniques, is a promising direction.
  • Web-Focused Tools: Integrating web-focused tools like ZAP and OpenVAS would enhance the framework's capabilities.
  • Knowledge Base Updates: Continuously updating the knowledge base is crucial for maintaining high performance.

Summary updated

Abstract

Penetration testing and vulnerability assessment are essential industry practices for safeguarding computer systems. As cyber threats grow in scale and complexity, the demand for pentesting has surged, surpassing the capacity of human professionals to meet it effectively. With advances in AI, particularly Large Language Models (LLMs), there have been attempts to automate the pentesting process. However, existing tools such as PentestGPT are still semi-manual, requiring significant professional human interaction to conduct pentests. To this end, we propose a novel LLM agent-based framework, AutoPentester, which automates the pentesting process. Given a target IP, AutoPentester automatically conducts pentesting steps using common security tools in an iterative process. It can dynamically generate attack strategies based on the tool outputs from the previous iteration, mimicking the human pentester approach. We evaluate AutoPentester using Hack The Box and custom-made VMs, comparing the results with the state-of-the-art PentestGPT. Results show that AutoPentester achieves a 27.0% better subtask completion rate and 39.5% more vulnerability coverage with fewer steps. Most importantly, it requires significantly fewer human interactions and interventions compared to PentestGPT. Furthermore, we recruit a group of security industry professional volunteers for a user survey and perform a qualitative analysis to evaluate AutoPentester against industry practices and compare it with PentestGPT. On average, AutoPentester received a score of 3.93 out of 5 based on user reviews, which was 19.8% higher than PentestGPT.
BibTeX
@misc{ginige2025autopentester,
  author = {Yasod Ginige and Akila Niroshan and Sajal Jain and Ranga Rodrigo},
  title = {{AutoPentester: An LLM Agent-based Framework for Automated Pentesting}},
  year = {2025},
  doi = {10.1109/trustcom66490.2025.00026},
}
Figure