AttackSeqBench: Benchmarking the Capabilities of LLMs for Attack Sequences Understanding

Hong-Hao Ma, Javier Yong, Yunshan Ma, K. M. Chen, Anis Yusof, Zhenkai Liang, Ee‐Chien Chang

TL;DR

This paper introduces AttackSeqBench, a new benchmark designed to rigorously evaluate how well Large Language Models (LLMs) can understand and reason about complex cyber attack sequences described in real-world Cyber Threat Intelligence (CTI) reports. By focusing on the tactical, technical, and procedural aspects of these sequences, AttackSeqBench aims to uncover LLMs' limitations and guide future development for more effective automated cybersecurity analysis.

What Problem Does This Paper Address?

Cybersecurity professionals rely heavily on Cyber Threat Intelligence (CTI) reports to understand and defend against evolving threats. However, these reports are often lengthy, unstructured, and require significant manual effort to extract actionable insights, especially regarding the step-by-step progression of adversary actions (attack sequences). While LLMs show promise in various NLP tasks, their ability to deeply understand and reason about the complex, multi-stage nature of cyber attacks, including the dependencies between different techniques and procedures, remains largely unexplored and underexplored. This gap hinders the effective application of LLMs in automating CTI analysis and improving cybersecurity defenses.

Methods

The authors developed AttackSeqBench, a benchmark that addresses three key challenges:

  1. Extensibility: They created an automated pipeline to generate questions and answers from CTI reports, allowing the benchmark to be easily updated with new threat intelligence.
  2. Reasoning Scalability: They evaluated not only standard LLMs but also Large Reasoning Models (LRMs) and various post-training strategies to assess their capabilities on complex reasoning tasks.
  3. Domain-Specific Epistemic Expandability: They explored different benchmark settings (Zero-Shot, Context, and Retrieval Augmented Generation - RAG) to understand how LLMs leverage external knowledge and their own internal knowledge to reason about cybersecurity concepts.

The benchmark includes three task types: AttackSeqBench-Tactic (inferring the overall tactic), AttackSeqBench-Technique (inferring a specific technique), and AttackSeqBench-Procedure (determining the likelihood of a specific procedure).

Datasets

The primary dataset used is a collection of 408 real-world CTI reports. From these reports, the authors constructed the AttackSeqBench benchmark, which comprises automatically generated Question-Answering (Q&A) pairs covering tactical, technical, and procedural aspects of cyber attacks. This new benchmark is designed to be more comprehensive and representative of real-world threat intelligence than previous CTI-related benchmarks, which often relied on curated knowledge bases or limited CTI report samples.

Key Insights and Ideas

  • LLMs struggle with nuanced attack sequence reasoning: Despite advancements, LLMs often fail to grasp the precise temporal and logical dependencies within attack sequences, a critical aspect of cybersecurity.
  • Context is king, but RAG needs refinement: Providing LLMs with contextual information (e.g., surrounding attack steps) significantly improves performance, but current RAG approaches don't always yield better results, suggesting a need for more sophisticated retrieval and integration methods.
  • LRMs don't always outperform LLMs in this domain: Unlike in general reasoning tasks, LRMs did not consistently outperform LLMs on AttackSeqBench, sometimes even performing worse, highlighting the domain-specific nature of the challenge.
  • Domain-specific knowledge is crucial: LLMs need specialized cybersecurity knowledge to effectively analyze CTI reports, and post-training strategies show promise but require further development.
  • Human evaluation remains a gold standard: While automatic evaluation metrics are useful, human experts still provide a more accurate assessment of LLM performance on complex cybersecurity reasoning tasks.

Key Results

  • Performance varies significantly: LLMs generally performed better in the Context setting than in Zero-Shot or RAG settings, especially for larger models.
  • AttackSeqBench-Tactic is the most challenging: This task showed the highest variance in performance across models, indicating that inferring high-level tactics from sequences is particularly difficult.
  • GPT-40 led performance: Among the evaluated models, GPT-40 generally achieved the highest accuracy across tasks and settings.
  • Post-training strategies improve performance: Techniques like SFT, RD, RLIF, and RLVR showed improvements, particularly in Zero-Shot settings, but still lagged behind models with task-adapted prompts.
  • Embedding models matter for RAG: While different embedding models showed comparable performance, domain-specific models like ATT&CK-BERT showed promise, even with fewer parameters, suggesting the importance of specialized knowledge.

Limitations and Future Work

  • Limited evaluation scope: The current benchmark primarily uses multiple-choice and Yes/No questions, which may not fully capture the depth of LLM reasoning.
  • Basic RAG and post-training implementations: The RAG and post-training strategies used were relatively basic and could be further refined.
  • Dataset size: While larger than previous benchmarks, the 408 CTI reports could be expanded for even broader coverage.

Future work includes:

  • Expanding evaluation to more complex tasks like reasoning and completion.
  • Developing more advanced RAG approaches and post-training strategies tailored for cybersecurity.
  • Continuously updating the CTI corpus to ensure the benchmark remains relevant and comprehensive.

Summary updated

Abstract

Cyber Threat Intelligence (CTI) reports document observations of cyber threats, synthesizing evidence about adversaries' actions and intent into actionable knowledge that informs detection, response, and defense planning. However, the unstructured and verbose nature of CTI reports poses significant challenges for security practitioners to manually extract and analyze such sequences. Although large language models (LLMs) exhibit promise in cybersecurity tasks such as entity extraction and knowledge graph construction, their understanding and reasoning capabilities towards behavioral sequences remains underexplored. To address this, we introduce AttackSeqBench, a benchmark designed to systematically evaluate LLMs' reasoning abilities across the tactical, technical, and procedural dimensions of adversarial behaviors, while satisfying Extensibility, Reasoning Scalability, and Domain-dpecific Epistemic Expandability. We further benchmark 7 LLMs, 5 LRMs and 4 post-training strategies across 3 benchmark settings and 3 benchmark tasks within our AttackSeqBench to identify their advantages and limitations in such specific domain. Our findings contribute to a deeper understanding of LLM-driven CTI report understanding and foster its application in cybersecurity operations. Our code of benchmark construction and evaluation and the corresponding dataset are available at: https://github.com/hulkima/AttackSeqBench.
BibTeX
@misc{ma2025attackseqbench,
  author = {Hong-Hao Ma and Javier Yong and Yunshan Ma and K. M. Chen and Anis Yusof and Zhenkai Liang and Ee‐Chien Chang},
  title = {{AttackSeqBench: Benchmarking the Capabilities of LLMs for Attack Sequences Understanding}},
  year = {2025},
  eprint = {2503.03170},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.03170},
}
Figure