AttackSeqBench: Benchmarking Large Language Models' Understanding of Sequential Patterns in Cyber Attacks

Javier Yong, Haokai Ma, Yunshan Ma, Anis Yusof, Zhenkai Liang, Ee-Chien Chang

Abstract

The observations documented in Cyber Threat Intelligence (CTI) reports play a critical role in describing adversarial behaviors, providing valuable insights for security practitioners to respond to evolving threats. Recent advancements of Large Language Models (LLMs) have demonstrated significant potential in various cybersecurity applications, including CTI report understanding and attack knowledge graph construction. While previous works have proposed benchmarks that focus on the CTI extraction ability of LLMs, the sequential characteristic of adversarial behaviors within CTI reports remains largely unexplored, which holds considerable significance in developing a comprehensive understanding of how adversaries operate. To address this gap, we introduce AttackSeqBench, a benchmark tailored to systematically evaluate LLMs' capability to understand and reason attack sequences in CTI reports. Our benchmark encompasses three distinct Question Answering (QA) tasks, each task focuses on the varying granularity in adversarial behavior. To alleviate the laborious effort of QA construction, we carefully design an automated dataset construction pipeline to create scalable and well-formulated QA datasets based on real-world CTI reports. To ensure the quality of our dataset, we adopt a hybrid approach of combining human evaluation and systematic evaluation metrics. We conduct extensive experiments and analysis with both fast-thinking and slow-thinking LLMs, while highlighting their strengths and limitations in analyzing the sequential patterns in cyber attacks. The overarching goal of this work is to provide a benchmark that advances LLM-driven CTI report understanding and fosters its application in real-world cybersecurity operations.

Introduction

Cyber Threat Intelligence (CTI) reports are essential for documenting adversarial behaviors. While Large Language Models (LLMs) have shown promise in extracting entities from these reports, their ability to grasp the sequential nature of attacks—how one step logically leads to another—remains under-explored. Understanding these dependencies is critical for predicting an attacker's next move. Existing benchmarks often focus on static knowledge or simple classification, failing to measure if an LLM understands that Step A must precede Step B in an Advanced Persistent Threat (APT) campaign.

Visualization of a multi-stage attack sequence and the AttackSeqBench concept.

Methods

To address this gap, we developed AttackSeqBench, an automated pipeline that transforms unstructured CTI reports into a structured benchmark. The framework consists of three main phases:

  1. Attack Sequence Construction: Reports are parsed into sequences of Tactics, Techniques, and Procedures (TTPs) mapped to the industry-standard MITRE ATT&CK framework.
  2. Automated Q&A Generation: Using GPT-4o, the system generates Multiple-Choice and Yes/No questions requiring "abductive reasoning" to infer missing steps in a sequence.
  3. Self-Refinement & Evaluation: A "Verifier" model checks the generated questions for clarity, logic, and consistency. Models are then evaluated across three settings: Zero-Shot, Context-based (using masked outlines), and RAG-empowered (using the MITRE ATT&CK knowledge base).
The AttackSeqBench framework, illustrating sequence construction, Q&A generation with self-refinement, and evaluation criteria.

Datasets

The benchmark utilizes a curated dataset of 408 CTI reports from major security vendors like Cisco Talos and Microsoft. This resulted in a large-scale dataset comprising:

  • AttackSeqBench-Tactic: 1,697 questions on high-level goals.
  • AttackSeqBench-Technique: 1,917 questions on specific methods.
  • AttackSeqBench-Procedure: 2,635 questions on granular actions.

This dataset significantly exceeds the scale of previous CTI benchmarks, which often relied on fewer than 100 reports.

Key Insights and Results

Our experiments revealed several critical findings regarding LLM performance in cybersecurity reasoning:

  • The "Overthinking" Trap: Reasoning models like DeepSeek-R1 did not always outperform standard LLMs. They often "overthought" temporal constraints, focusing on general plausibility rather than the specific sequence provided in the report.
  • RAG Limitations: Retrieval-Augmented Generation (RAG) sometimes decreased performance. Retrieved snippets acted as noise, causing models to over-rely on external definitions instead of the provided attack flow.
  • Contextual Advantage: Models performed best when provided with a "masked CTI outline," suggesting they excel at "filling in the blanks" when a structural map is available.
  • Performance Benchmarks: Qwen2.5-32B and GPT-4o were the top performers. While performance generally scaled with model size, standard LLMs often beat reasoning-specialized models on temporal logic tasks. The best models reached accuracy levels of 0.60–0.70 in context-aware settings, comparable to the human expert baseline of 0.63.

Conclusion

AttackSeqBench provides a systematic way to evaluate LLMs' understanding of attack sequences. Our findings highlight that while LLMs are becoming proficient at technical identification, reasoning through the chronological and tactical intent of an attack remains a significant challenge. Future work will focus on more complex reasoning tasks and developing "sequence-aware" retrieval systems to better support cybersecurity operations.

Summary updated

Abstract

The observations documented in Cyber Threat Intelligence (CTI) reports play a critical role in describing adversarial behaviors, providing valuable insights for security practitioners to respond to evolving threats. Recent advancements of Large Language Models (LLMs) have demonstrated significant potential in various cybersecurity applications, including CTI report understanding and attack knowledge graph construction. While previous works have proposed benchmarks that focus on the CTI extraction ability of LLMs, the sequential characteristic of adversarial behaviors within CTI reports remains largely unexplored, which holds considerable significance in developing a comprehensive understanding of how adversaries operate. To address this gap, we introduce AttackSeqBench, a benchmark tailored to systematically evaluate LLMs' capability to understand and reason attack sequences in CTI reports. Our benchmark encompasses three distinct Question Answering (QA) tasks, each task focuses on the varying granularity in adversarial behavior. To alleviate the laborious effort of QA construction, we carefully design an automated dataset construction pipeline to create scalable and well-formulated QA datasets based on real-world CTI reports. To ensure the quality of our dataset, we adopt a hybrid approach of combining human evaluation and systematic evaluation metrics. We conduct extensive experiments and analysis with both fast-thinking and slow-thinking LLMs, while highlighting their strengths and limitations in analyzing the sequential patterns in cyber attacks. The overarching goal of this work is to provide a benchmark that advances LLM-driven CTI report understanding and fosters its application in real-world cybersecurity operations. Our dataset and code are available at https://github.com/Javiery3889/AttackSeqBench .
BibTeX
@misc{yong2025attackseqbench,
  author = {Javier Yong and Haokai Ma and Yunshan Ma and Anis Yusof and Zhenkai Liang and Ee-Chien Chang},
  title = {{AttackSeqBench: Benchmarking Large Language Models' Understanding of Sequential Patterns in Cyber Attacks}},
  year = {2025},
  eprint = {2503.03170},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.03170},
}
Figure