TL;DR
This paper introduces ThreatKG, an AI-powered system that automates the collection and analysis of open-source cyber threat intelligence (OSCTI) from unstructured text reports. By building a comprehensive knowledge graph, ThreatKG aims to provide a richer, more connected understanding of threats beyond simple IOCs, enabling more robust defensive strategies.
What Problem Does This Paper Address?
The cybersecurity landscape is constantly evolving, with sophisticated attacks becoming more prevalent. While open-source cyber threat intelligence (OSCTI) is crucial for staying ahead, current methods primarily rely on structured "Indicators of Compromise" (IOCs) like IP addresses or file hashes. These IOCs are low-level and isolated, offering only a fragmented view of an attack. The vast amount of valuable, interconnected information hidden within unstructured OSCTI reports (like security articles and threat analyses) remains largely untapped. This gap limits our ability to understand complex, multi-step attack behaviors and develop truly resilient defenses. ThreatKG aims to bridge this gap by extracting and organizing this rich, unstructured knowledge.
Methods
ThreatKG employs a three-phase architecture:
- OSCTI Report Collection: A robust, multi-threaded crawler framework gathers reports from diverse security websites and repositories (e.g., threat encyclopedias, security blogs).
- Threat Knowledge Extraction: This phase involves several key steps:
- Parsing and Relevance Checking: Reports are parsed into a unified format (UTKR). AI-based checkers are trained to filter out irrelevant content (like advertisements) using features like keyword density and TF-IDF.
- Entity Extraction: A deep learning-based approach using a Bidirectional LSTM-CRF (BiLSTM-CRF) model is employed for Neural Named Entity Recognition (NER) to identify threat entities (malware, threat actors, TTPs, etc.). Special handling is included for security-specific nuances in text.
- Relation Extraction: Two methods are used: dependency parsing for verb-based relations and a Piecewise Convolutional Neural Network with attention (PCNN-ATT) for neural relation extraction, identifying how entities interact. Coreference resolution is used to link related mentions.
- Data Programming: To overcome the scarcity of labeled data for training deep learning models, ThreatKG leverages data programming (specifically, the Snorkel framework) to synthesize annotations from noisy labeling functions, significantly improving model performance.
- Threat Knowledge Graph Construction: Extracted entities and relations are organized into a hierarchical threat knowledge ontology and stored in a Neo4j graph database, forming the Threat Knowledge Graph (TKG). This graph is continuously updated with new intelligence.
The system is designed to be modular and extensible, allowing for easy integration of new data sources and knowledge types.
Datasets
- Training and Evaluation Data: The authors created a ground-truth labeled dataset of 141 OSCTI reports from seven different sources (APTnotes, Kaspersky Threat Encyclopedia, Symantec Threat Encyclopedia, and four enterprise security blogs). This dataset was manually annotated for entities and relations.
- New Benchmarks: While not explicitly stated as a "new benchmark" in the traditional sense, the curated and annotated dataset of 141 reports serves as a valuable resource for evaluating OSCTI extraction systems.
Key Insights and Ideas
- The Value of Unstructured Data: The paper strongly emphasizes that the real richness of OSCTI lies not just in structured IOCs but in the narrative and contextual information within unstructured text.
- Hierarchical Ontology for Comprehensive Modeling: A multi-layered ontology (report context, threat behavior, threat context) is crucial for capturing threat intelligence at different granularities and interconnections.
- Data Programming is a Game-Changer for Scarce Data: The significant performance improvement in relation extraction after using data programming highlights its effectiveness in addressing the common challenge of limited labeled data in specialized domains like cybersecurity.
- Source-Specific Classifiers Outperform Universal Ones for Relevance Filtering: The distributional shift across different OSCTI sources necessitates tailored models for accurate filtering of irrelevant reports.
- Extensibility is Key: A modular architecture is vital for adapting to the ever-changing landscape of OSCTI sources and formats.
Key Results
- Report Filtering: Source-specific checkers achieved high performance, with average F1 scores above 86% and low false negative rates (FNRs) below 3.77%, indicating effective filtering of irrelevant reports.
- Relation Extraction: ThreatKG achieved an aggregated F1 score of 85% for relation extraction, a significant improvement from 79% without data programming. This improvement was particularly notable for relation types with fewer initial training instances.
- Comparison with Baselines: ThreatKG outperformed TTPDrill and EXTRACTOR in relation extraction, demonstrating its broader scope and higher accuracy, especially in capturing relations beyond simple IOC interactions.
- Runtime Performance: The system is efficient enough for real-world deployment, processing approximately 24.7 reports per minute. Dependency parsing was identified as a bottleneck, particularly for longer reports.
Limitations and Future Work
- Limitations:
- The performance of universal checkers was worse than source-specific ones, indicating the challenge of distributional shift.
- Dependency parsing for long content OSCTI reports is time-consuming, impacting overall extraction speed.
- The current implementation focuses on entities co-occurring within the same sentence for relation extraction.
- Future Work:
- Explore other types of security applications that can be built upon the threat knowledge graph, such as intrusion detection and cyber threat hunting.
- Investigate methods to further improve the efficiency of dependency parsing.
- Expand relation extraction to consider entities across different sentences.
Summary updated