An AI-Powered System for Automated Open-Source Cyber Threat Intelligence Gathering and Management

Peng Gao, Xiaoyuan Liu, Edward Choi, Sibo Ma, Xinyu Yang, Dawn SongLAMPS@CCS 2024 · OpenReview

TL;DR

This paper introduces ThreatKG, an AI-powered system that automates the collection and analysis of open-source cyber threat intelligence (OSCTI) from unstructured text reports. By building a comprehensive knowledge graph, ThreatKG aims to provide a richer, more connected understanding of threats beyond simple IOCs, enabling more robust defensive strategies.

What Problem Does This Paper Address?

The cybersecurity landscape is constantly evolving, with sophisticated attacks becoming more prevalent. While open-source cyber threat intelligence (OSCTI) is crucial for staying ahead, current methods primarily rely on structured "Indicators of Compromise" (IOCs) like IP addresses or file hashes. These IOCs are low-level and isolated, offering only a fragmented view of an attack. The vast amount of valuable, interconnected information hidden within unstructured OSCTI reports (like security articles and threat analyses) remains largely untapped. This gap limits our ability to understand complex, multi-step attack behaviors and develop truly resilient defenses. ThreatKG aims to bridge this gap by extracting and organizing this rich, unstructured knowledge.

Methods

ThreatKG employs a three-phase architecture:

  1. OSCTI Report Collection: A robust, multi-threaded crawler framework gathers reports from diverse security websites and repositories (e.g., threat encyclopedias, security blogs).
  2. Threat Knowledge Extraction: This phase involves several key steps:
  3. Parsing and Relevance Checking: Reports are parsed into a unified format (UTKR). AI-based checkers are trained to filter out irrelevant content (like advertisements) using features like keyword density and TF-IDF.
  4. Entity Extraction: A deep learning-based approach using a Bidirectional LSTM-CRF (BiLSTM-CRF) model is employed for Neural Named Entity Recognition (NER) to identify threat entities (malware, threat actors, TTPs, etc.). Special handling is included for security-specific nuances in text.
  5. Relation Extraction: Two methods are used: dependency parsing for verb-based relations and a Piecewise Convolutional Neural Network with attention (PCNN-ATT) for neural relation extraction, identifying how entities interact. Coreference resolution is used to link related mentions.
  6. Data Programming: To overcome the scarcity of labeled data for training deep learning models, ThreatKG leverages data programming (specifically, the Snorkel framework) to synthesize annotations from noisy labeling functions, significantly improving model performance.
  7. Threat Knowledge Graph Construction: Extracted entities and relations are organized into a hierarchical threat knowledge ontology and stored in a Neo4j graph database, forming the Threat Knowledge Graph (TKG). This graph is continuously updated with new intelligence.

The system is designed to be modular and extensible, allowing for easy integration of new data sources and knowledge types.

Datasets

  • Training and Evaluation Data: The authors created a ground-truth labeled dataset of 141 OSCTI reports from seven different sources (APTnotes, Kaspersky Threat Encyclopedia, Symantec Threat Encyclopedia, and four enterprise security blogs). This dataset was manually annotated for entities and relations.
  • New Benchmarks: While not explicitly stated as a "new benchmark" in the traditional sense, the curated and annotated dataset of 141 reports serves as a valuable resource for evaluating OSCTI extraction systems.

Key Insights and Ideas

  • The Value of Unstructured Data: The paper strongly emphasizes that the real richness of OSCTI lies not just in structured IOCs but in the narrative and contextual information within unstructured text.
  • Hierarchical Ontology for Comprehensive Modeling: A multi-layered ontology (report context, threat behavior, threat context) is crucial for capturing threat intelligence at different granularities and interconnections.
  • Data Programming is a Game-Changer for Scarce Data: The significant performance improvement in relation extraction after using data programming highlights its effectiveness in addressing the common challenge of limited labeled data in specialized domains like cybersecurity.
  • Source-Specific Classifiers Outperform Universal Ones for Relevance Filtering: The distributional shift across different OSCTI sources necessitates tailored models for accurate filtering of irrelevant reports.
  • Extensibility is Key: A modular architecture is vital for adapting to the ever-changing landscape of OSCTI sources and formats.

Key Results

  • Report Filtering: Source-specific checkers achieved high performance, with average F1 scores above 86% and low false negative rates (FNRs) below 3.77%, indicating effective filtering of irrelevant reports.
  • Relation Extraction: ThreatKG achieved an aggregated F1 score of 85% for relation extraction, a significant improvement from 79% without data programming. This improvement was particularly notable for relation types with fewer initial training instances.
  • Comparison with Baselines: ThreatKG outperformed TTPDrill and EXTRACTOR in relation extraction, demonstrating its broader scope and higher accuracy, especially in capturing relations beyond simple IOC interactions.
  • Runtime Performance: The system is efficient enough for real-world deployment, processing approximately 24.7 reports per minute. Dependency parsing was identified as a bottleneck, particularly for longer reports.

Limitations and Future Work

  • Limitations:
    • The performance of universal checkers was worse than source-specific ones, indicating the challenge of distributional shift.
  • Dependency parsing for long content OSCTI reports is time-consuming, impacting overall extraction speed.
  • The current implementation focuses on entities co-occurring within the same sentence for relation extraction.
  • Future Work:
    • Explore other types of security applications that can be built upon the threat knowledge graph, such as intrusion detection and cyber threat hunting.
  • Investigate methods to further improve the efficiency of dependency parsing.
  • Expand relation extraction to consider entities across different sentences.

Summary updated

Abstract

Open-source cyber threat intelligence (OSCTI) has become essential for keeping up with the rapidly changing threat landscape. However, current OSCTI gathering and management solutions mainly focus on structured Indicators of Compromise (IOC) feeds, which are low-level and isolated, providing only a narrow view of potential threats. Meanwhile, the extensive and interconnected knowledge found in the unstructured text of numerous OSCTI reports (e.g., security articles, threat reports) available publicly is still largely underexplored.To bridge the gap, we propose ThreatKG, an automated system for OSCTI gathering and management. ThreatKG efficiently collects a large number of OSCTI reports from multiple sources, leverages specialized AI-based techniques to extract high-quality knowledge about various threat entities and their relationships, and constructs and continuously updates a threat knowledge graph by integrating new OSCTI data. ThreatKG features a modular and extensible design, allowing for the addition of components to accommodate diverse OSCTI report structures and knowledge types. Our extensive evaluations demonstrate ThreatKG's practical effectiveness in enhancing threat knowledge gathering and management.
BibTeX
@inproceedings{gao2024ai,
  author = {Peng Gao and Xiaoyuan Liu and Edward Choi and Sibo Ma and Xinyu Yang and Dawn Song},
  title = {{An AI-Powered System for Automated Open-Source Cyber Threat Intelligence Gathering and Management}},
  booktitle = {LAMPS@CCS},
  year = {2024},
  url = {https://openreview.net/forum?id=suuhyG5EeJ},
}
Figure