Full-Stack Information Extraction System for Cybersecurity Intelligence

Youngja Park, Taesung LeeEMNLP 2022

TL;DR

Cybersecurity analysts are drowning in a sea of threat intelligence reports. This paper introduces SecIE, a novel full-stack system that automatically extracts key entities (like malware, attackers, and vulnerabilities) and their relationships from these reports. SecIE achieves impressive accuracy, significantly aiding analysts in quickly understanding and responding to emerging cyber threats.

What Problem Does This Paper Address?

The digital landscape is under constant attack, with cyber threats evolving at an alarming rate. To combat this, security analysts rely on Cyber Threat Intelligence (CTI) reports, which are published daily. However, the sheer volume and complexity of these reports make it incredibly difficult for analysts to digest the crucial information in a timely manner. These reports contain a wealth of interconnected details about attackers, victims, tools, indicators of compromise (IOCs), and the relationships between them. Manually sifting through this data is time-consuming and prone to errors, hindering rapid response to ongoing threats. This paper aims to bridge this gap by providing an automated system to extract and structure this vital intelligence.

Methods

SecIE employs a sophisticated pipeline architecture to process CTI reports. The core components include:

  • Document Parsing: Extracts text content and structural information (like titles, links, and tables) using Apache Tika.
  • Linguistic Analysis: Performs sentence boundary detection, part-of-speech tagging, and dependency parsing. A key innovation here is a domain-specific customization to improve parsing accuracy for security-related text, which often contains complex entity names and unusual structures.
  • Entity Extraction: Identifies 26 distinct entity types, categorized based on the STIX standard. This is achieved through a combination of:
  • Pattern-based methods: For entities with well-defined patterns (e.g., IP addresses, email addresses), including support for obfuscated IOC patterns.
  • Dictionary-based methods: For known entities like malware families or threat actors.
  • Lexico-syntactic pattern matching: To discover new entity mentions based on linguistic patterns.
  • Classification-based extraction: For entities like AvSignatures that don't fit neat patterns, using machine learning with character n-grams and word shape features.
  • Coreference Resolution: Links mentions of the same entity within and across sentences (e.g., recognizing that "the new ransomware" refers to "Trojan-Ransom.Win32.Xpan").
  • Topic Entity Detection: Identifies the central focus of a CTI report, which is crucial for understanding the context of other extracted information.
  • Relation Extraction: Identifies relationships between entities using:
  • OpenIE Relation Extraction: Focuses on specific syntactic structures involving verbs and noun phrases.
  • Cooccurrence-based Relation Extraction: Identifies relationships when two security entities frequently appear together, even if not directly linked by a verb.
  • Relations with Topic Entity: Connects entities to the main topic entity of the report.
  • Relation Time Assignment: Extracts and normalizes temporal information associated with relations, which is critical for understanding the timeline of cyber events.

Datasets

The system was evaluated on a manually labeled dataset of 133 CTI reports, containing 6,438 sentences and 108,021 tokens. This dataset was annotated by three security experts and two professional annotators over five months. The final dataset used for evaluation consisted of 3,295 entity mentions and 1,216 relation mentions, with labels only included if all annotators agreed.

Key Insights and Ideas

  • Domain-Specific NLP is Crucial: Standard NLP tools struggle with the unique language and structures found in cybersecurity reports. SecIE's success hinges on its domain customization, particularly in parsing and handling obfuscated IOCs.
  • The Power of a Full-Stack Approach: By integrating entity extraction, coreference resolution, and relation extraction into a cohesive pipeline, SecIE can build a richer, more interconnected understanding of threat intelligence than individual components could achieve.
  • Temporal Information is Key: The ability to extract and link temporal information to relations is highlighted as critical for cybersecurity investigations, providing a timeline of events.
  • Unsupervised/Semi-Supervised Methods are Necessary: The difficulty and cost of creating large, high-quality labeled datasets for cybersecurity mean that unsupervised or semi-supervised techniques are essential for building robust systems.

Key Results

  • Entity Extraction: SecIE achieves an impressive 92.2% F1-score on mention-level entity extraction across all entity types. For specific entity types, performance is even higher, with 100% F1 for Hash and Network entities.
  • Relation Extraction: The system achieves approximately 70% F1-score for relation extraction, with performance improving when coreference resolution is included.
  • Comparison to BERT: SecIE significantly outperforms a BERT-based model on entity extraction for the 13 test documents, demonstrating the effectiveness of its domain-specific approach.

Limitations and Future Work

  • Data Annotation Challenges: The paper acknowledges the significant effort required to create high-quality labeled data for the cybersecurity domain, noting that even with expert annotators, only a limited number of documents could be annotated.
  • Future Work: The authors suggest that SecIE can be used for downstream applications like building knowledge graphs, identifying inconsistencies in CVE databases, and providing enhanced insights for security analysts. The system's ability to transform unstructured text into structured formats makes it a valuable asset for various security tools and workflows.

Summary updated

BibTeX
@inproceedings{park2022full,
  author = {Youngja Park and Taesung Lee},
  title = {{Full-Stack Information Extraction System for Cybersecurity Intelligence}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
}
Figure