TL;DR
ExCyTIn-Bench is the first standardized benchmark designed to evaluate LLM agents on the complex, multi-hop task of cyber threat investigation. By providing an interactive SQL environment filled with real-world security logs and questions anchored to explicit incident graphs, it shifts the evaluation of cybersecurity LLMs from passive knowledge memorization to active, relational reasoning. Experiments show that even the most advanced frontier models leave significant headroom for improvement, peaking at a score of 0.606.
What Problem Does This Paper Address?
When a cyberattack occurs, Security Operations Center (SOC) analysts don't just answer trivia questions; they act as digital detectives. They must sift through millions of heterogeneous logs (endpoint activities, sign-ins, emails), map out connections between entities (like IP addresses and file hashes), and reconstruct multi-stage attack chains.
While Large Language Models (LLMs) possess vast amounts of cybersecurity knowledge, existing benchmarks primarily evaluate them on static knowledge retrieval (e.g., multiple-choice Q&A about threat intelligence). There is a distinct lack of frameworks to test LLM agents acting in realistic, interactive environments where they must generate queries, discover database schemas, and actively investigate an incident.
Methods
The authors introduce ExCyTIn-Bench, which evaluates agents through a three-step framework:
- Data Collection: The data originates from a fictional but realistically configured Microsoft Azure tenant ("Alpine Ski House"). It contains 57 distinct log tables capturing real-world attack chains.
- Graph-Based Question Generation: To avoid generating generic or non-deterministic questions, the authors construct a bipartite incident graph for each attack scenario, mapping Alert vertices to Entity vertices. An LLM uses pairs of distant nodes on this graph to generate realistic security questions, using the starting node as context and the end node as the target answer.
- Interactive Environment: The benchmark sets up a MySQL Docker environment. The agent receives a question, explores the database by executing SQL queries (actions), receives query results (observations), and eventually submits an answer.
To accurately score agents, the benchmark implements a fine-grained Progress Reward Calculation. If an agent fails to find the exact final answer, it can still receive partial credit if its submitted reasoning contains intermediate steps (Indicators of Compromise) discovered along the gold-standard graph path.
Datasets
ExCyTIn-Bench consists of 7,542 generated questions derived from 8 distinct multi-stage attack scenarios (such as the "Manatee Tempest Ransomware" campaign and "Business Email Compromise"). The authors curated a diverse test set of 589 questions for evaluation.
Compared to standard text-to-SQL benchmarks like Spider, ExCyTIn-Bench is massive and highly complex, averaging 45 tables and over 364K rows per incident database.
Key Insights and Ideas
- Investigation is Inherently Relational: Tracing a cyberattack is best modeled as a walk on a bipartite graph. Anchoring questions to explicit graph paths provides an automated, explainable ground truth.
- The Danger of Naive Fine-Tuning: When the authors fine-tuned GPT-4o on successful investigation trajectories, its performance on unseen test incidents drastically dropped (e.g., from 0.293 to 0.241 on one incident). This indicates that standard supervised fine-tuning causes the model to overfit to specific attack contexts rather than learning generalized investigation skills.
- Alert Logs are Crucial Filters: In an ablation study, removing alert tables (leaving only raw system logs) caused a significant performance drop. This highlights that expert-crafted detection rules are vital for filtering out background noise, even for advanced AI agents.
- Reasoning Models Scale with Interaction: Explicit reasoning models (like
o4-mini) show a strong capacity to scale performance as the number of allowed interaction turns increases, outperforming traditional chat models at higher turn counts.
Key Results
The benchmark proved to be highly challenging for all evaluated models:
- Top Performers:
Claude-Opus-4.5achieved the highest overall reward of 0.606, followed closely byGPT-5.1 (Reasoning=High)at 0.582. - Open-Source Progress: Open-source models are rapidly closing the gap.
Llama4-Mav-17Bachieved a reward of 0.290, performing on par with proprietary chat models likeGPT-4o(0.293). - Prompting Frameworks Matter: Implementing advanced agent frameworks like ReAct combined with Self-Reflection or rule-distillation (Expel) significantly boosted performance over the baseline prompt. For instance, ReAct+Reflect raised GPT-4o's single-trial score from 0.260 to 0.563.
Limitations and Future Work
While highly rigorous, ExCyTIn-Bench currently relies on an LLM-as-judge (GPT-4o) to evaluate final answers due to the flexible formatting of agent outputs (though human validation confirmed 100% alignment on a sample subset). Additionally, the data is sourced from a single enterprise tenant setup, albeit across eight distributionally different attack scenarios.
Given that standard fine-tuning led to severe overfitting, the authors note that the fine-grained process rewards provided by ExCyTIn-Bench make it an ideal testbed for training future agents using Reinforcement Learning with Value Regularization (RLVR) to incentivize genuine reasoning over memorization.
Summary updated