ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation

Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel Raúl Meléndez Luján, Srisuma Movva, Yogesh K Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, Julia Kiseleva, Anand Mudgerikar

TL;DR

ExCyTIn-Bench is the first standardized benchmark designed to evaluate LLM agents on the complex, multi-hop task of cyber threat investigation. By providing an interactive SQL environment filled with real-world security logs and questions anchored to explicit incident graphs, it shifts the evaluation of cybersecurity LLMs from passive knowledge memorization to active, relational reasoning. Experiments show that even the most advanced frontier models leave significant headroom for improvement, peaking at a score of 0.606.

What Problem Does This Paper Address?

When a cyberattack occurs, Security Operations Center (SOC) analysts don't just answer trivia questions; they act as digital detectives. They must sift through millions of heterogeneous logs (endpoint activities, sign-ins, emails), map out connections between entities (like IP addresses and file hashes), and reconstruct multi-stage attack chains.

While Large Language Models (LLMs) possess vast amounts of cybersecurity knowledge, existing benchmarks primarily evaluate them on static knowledge retrieval (e.g., multiple-choice Q&A about threat intelligence). There is a distinct lack of frameworks to test LLM agents acting in realistic, interactive environments where they must generate queries, discover database schemas, and actively investigate an incident.

Methods

The authors introduce ExCyTIn-Bench, which evaluates agents through a three-step framework:

  1. Data Collection: The data originates from a fictional but realistically configured Microsoft Azure tenant ("Alpine Ski House"). It contains 57 distinct log tables capturing real-world attack chains.
  2. Graph-Based Question Generation: To avoid generating generic or non-deterministic questions, the authors construct a bipartite incident graph for each attack scenario, mapping Alert vertices to Entity vertices. An LLM uses pairs of distant nodes on this graph to generate realistic security questions, using the starting node as context and the end node as the target answer.
  3. Interactive Environment: The benchmark sets up a MySQL Docker environment. The agent receives a question, explores the database by executing SQL queries (actions), receives query results (observations), and eventually submits an answer.
Figure 1. Overview of constructing ExCyTin-Bench. Left Triangle: A complete Azure tenant with security logs, alerts and incidents. Middle Square: The ExCyTin playground, with the questions, environment and evaluation ready for evaluating agents.

To accurately score agents, the benchmark implements a fine-grained Progress Reward Calculation. If an agent fails to find the exact final answer, it can still receive partial credit if its submitted reasoning contains intermediate steps (Indicators of Compromise) discovered along the gold-standard graph path.

Datasets

ExCyTIn-Bench consists of 7,542 generated questions derived from 8 distinct multi-stage attack scenarios (such as the "Manatee Tempest Ransomware" campaign and "Business Email Compromise"). The authors curated a diverse test set of 589 questions for evaluation.

Compared to standard text-to-SQL benchmarks like Spider, ExCyTIn-Bench is massive and highly complex, averaging 45 tables and over 364K rows per incident database.

Figure 3. Example Question Generation. The start alert and entities will be used as context, and the question asks about the end entity (ID 20). A solution path is also based on the graph.

Key Insights and Ideas

  • Investigation is Inherently Relational: Tracing a cyberattack is best modeled as a walk on a bipartite graph. Anchoring questions to explicit graph paths provides an automated, explainable ground truth.
  • The Danger of Naive Fine-Tuning: When the authors fine-tuned GPT-4o on successful investigation trajectories, its performance on unseen test incidents drastically dropped (e.g., from 0.293 to 0.241 on one incident). This indicates that standard supervised fine-tuning causes the model to overfit to specific attack contexts rather than learning generalized investigation skills.
  • Alert Logs are Crucial Filters: In an ablation study, removing alert tables (leaving only raw system logs) caused a significant performance drop. This highlights that expert-crafted detection rules are vital for filtering out background noise, even for advanced AI agents.
  • Reasoning Models Scale with Interaction: Explicit reasoning models (like o4-mini) show a strong capacity to scale performance as the number of allowed interaction turns increases, outperforming traditional chat models at higher turn counts.

Key Results

The benchmark proved to be highly challenging for all evaluated models:

  • Top Performers: Claude-Opus-4.5 achieved the highest overall reward of 0.606, followed closely by GPT-5.1 (Reasoning=High) at 0.582.
  • Open-Source Progress: Open-source models are rapidly closing the gap. Llama4-Mav-17B achieved a reward of 0.290, performing on par with proprietary chat models like GPT-4o (0.293).
  • Prompting Frameworks Matter: Implementing advanced agent frameworks like ReAct combined with Self-Reflection or rule-distillation (Expel) significantly boosted performance over the baseline prompt. For instance, ReAct+Reflect raised GPT-4o's single-trial score from 0.260 to 0.563.
Reward vs. Cost Pareto Front for various models

Limitations and Future Work

While highly rigorous, ExCyTIn-Bench currently relies on an LLM-as-judge (GPT-4o) to evaluate final answers due to the flexible formatting of agent outputs (though human validation confirmed 100% alignment on a sample subset). Additionally, the data is sourced from a single enterprise tenant setup, albeit across eight distributionally different attack scenarios.

Given that standard fine-tuning led to severe overfitting, the authors note that the fine-grained process rewards provided by ExCyTIn-Bench make it an ideal testbed for training future agents using Reinforcement Learning with Value Regularization (RLVR) to incentivize genuine reasoning over memorization.

Summary updated

Abstract

We present ExCyTIn-Bench, the first benchmark to Evaluate an LLM agent X on the task of Cyber Threat Investigation through security questions derived from investigation graphs. Real-world security analysts must sift through a large number of heterogeneous security logs, follow multi-hop chains of evidence to investigate threats. With the developments of LLMs, building LLM-based agents for automatic threat investigation is a promising direction. We construct a benchmark from a controlled Azure tenant including a SQL environment covering 57 log tables from Microsoft Sentinel and related services, and 7542 generated questions. We leverage security logs extracted with expert-crafted detection logic to build threat investigation graphs, and then generate questions with LLMs using paired nodes on the graph, taking the start node as background context and the end node as answer. Anchoring each question to these explicit nodes and edges not only provides automatic, explainable ground truth answers but also makes the pipeline reusable and readily extensible to new logs. Our comprehensive experiments on the test set with different models confirm the difficulty of the task: the best model so far can achieve a reward of 0.606, leaving much headroom for future research. The code is available at https://github.com/microsoft/SecRL
BibTeX
@inproceedings{wu2026excytin,
  author = {Yiran Wu and Mauricio Velazco and Andrew Zhao and Manuel Raúl Meléndez Luján and Srisuma Movva and Yogesh K Roy and Quang Nguyen and Roberto Rodriguez and Qingyun Wu and Michael Albada and Julia Kiseleva and Anand Mudgerikar},
  title = {{ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=4pQVVSKdkH},
}
Figure