CyberSleuth: Autonomous Blue-Team LLM Agent for Web Attack Forensics

Stefano Fumero, Kai Huang, Matteo Boffa, Danilo Giordano, Marco Mellia, Zied Ben-Houidi, Dario Rossi

Abstract

Large Language Model (LLM) agents are powerful tools for automating complex tasks. In cybersecurity, researchers have primarily explored their use in red-team operations such as vulnerability discovery and penetration tests. Defensive uses for incident response and forensics have received comparatively less attention and remain at an early stage. This work presents a systematic study of LLM-agent design for the forensic investigation of realistic web application attacks. We propose CyberSleuth, an autonomous agent that processes packet-level traces and application logs to identify the targeted service, the exploited vulnerability (CVE), and attack success. We evaluate the consequences of core design decisions - spanning tool integration and agent architecture - and provide interpretable guidance for practitioners. We benchmark four agent architectures and six LLM backends on 20 incident scenarios of increasing complexity, identifying CyberSleuth as the best-performing design. In a separate set of 10 incidents from 2025, CyberSleuth correctly identifies the exact CVE in 80% of cases. At last, we conduct a human study with 22 experts, which rated the reports of CyberSleuth as complete, useful, and coherent. They also expressed a slight preference for DeepSeek R1, a good news for open source LLM. To foster progress in defensive LLM research, we release both our benchmark and the CyberSleuth platform as a foundation for fair, reproducible evaluation of forensic agents.
BibTeX
@misc{fumero2025cybersleuth,
  author = {Stefano Fumero and Kai Huang and Matteo Boffa and Danilo Giordano and Marco Mellia and Zied Ben-Houidi and Dario Rossi},
  title = {{CyberSleuth: Autonomous Blue-Team LLM Agent for Web Attack Forensics}},
  year = {2025},
  eprint = {2508.20643},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.20643},
}
Figure