CASIE: Extracting Cybersecurity Event Information from Text

Taneeya Satyapanich, Francis Ferraro, Tim FininAAAI 2020 · DOI

TL;DR

This paper introduces CASIE, a novel system for extracting detailed information about cybersecurity events from news text. By defining a new set of event types, roles, and arguments, and creating a richly annotated corpus, CASIE aims to build comprehensive knowledge graphs for better understanding and defense against cyber threats.

What Problem Does This Paper Address?

Cyberattacks and cybercrimes are increasingly frequent, sophisticated, and exploit new vulnerabilities. To effectively defend against these evolving threats, it's crucial to have a deep and up-to-date understanding of current trends, attack patterns, and vulnerabilities. Manually sifting through vast amounts of news and security reports to gather this information is time-consuming and prone to error. Existing event extraction systems often focus on general life events, lacking the domain-specific expertise and complexity required for cybersecurity. This paper addresses the challenge of automatically extracting structured information about cybersecurity events from unstructured text, enabling the creation of knowledge bases that can power automated defense systems and inform human analysts.

Methods

CASIE employs a multi-stage neural network architecture for event extraction. The core components include:

  • Event Nugget and Event Argument Detection: A Bidirectional LSTM (Bi-LSTM) network is used for both detecting event nuggets (key phrases indicating an event) and identifying event arguments (participants and properties of the event). This system leverages a rich set of linguistic features, including part-of-speech tags, named entity recognition, dependency relations, and shallow syntactic chunking, along with various word embeddings (Transfer-Word2vec, Domain-Word2vec, Cyber-Word2vec, and Pre-built BERT).
  • Event Argument and Role Linking: A fully-connected neural network assigns roles to identified event arguments based on the event type. This approach uses event type information to constrain role predictions, improving accuracy.
  • Event Realis Identification: A separate neural network classifies the realis value of an event (Actual, Other, or Generic), indicating whether the event actually happened, failed, or is being referenced conceptually.
  • Attention Mechanism: An attention layer is incorporated into the event argument detection system to help the model focus on the most relevant parts of the text.

Datasets

The paper introduces a new, challenging corpus of approximately 1,000 English news articles from 2017-2019. These articles were manually annotated with rich, event-based information, covering five specific cybersecurity event subtypes: Attack.Databreach, Attack.Phishing, Attack.Ransom, Discover.Vulnerability, and Patch.Vulnerability. This corpus is significantly larger and more complex than existing datasets for general event extraction. The annotations include event nuggets, event arguments, their types, and semantic roles.

Key Insights and Ideas

  • Domain-Specific Event Schema is Crucial: The paper highlights that cybersecurity events have unique complexities and require a specialized schema (event types, roles, arguments) distinct from general event extraction tasks.
  • Rich Linguistic Features Enhance Performance: Combining various linguistic features with deep learning models significantly improves the performance of both event nugget and argument detection.
  • Pre-trained BERT Embeddings Offer Advantages: For both event nugget and argument detection, pre-trained BERT embeddings generally yielded the best results, demonstrating the power of context-aware representations for this domain.
  • Event Type Context Improves Role Linking: Specifying the event type during argument role linking leads to higher performance, as it allows for more targeted and accurate role assignment.
  • Annotation Challenges are Significant: The paper acknowledges the inherent difficulties in annotating cybersecurity events, such as distinguishing between atomic actions within a larger event and resolving coreference across multiple mentions.

Key Results

  • High Performance Achieved: The CASIE system demonstrates strong performance across its components. For instance, nugget detection scores reached up to 79.94% F1 with Pre-built BERT, and argument detection also showed competitive results.
  • Feature Ablation Studies: Ablation studies confirmed the effectiveness of the proposed feature sets for both nugget and argument detection.
  • Role Linking Improvement: Specifying the event type for role linking improved scores, with some roles like VS-Owner showing significant gains (from 0.09 to 0.75).
  • Realis Identification Accuracy: The realis identification system achieved an F1 score of 77.65%.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that identifying "Capabilities" and "Purpose" arguments can be difficult due to their often long and technical nature.
  • The GPE argument type presented challenges due to its varied usage (government organization vs. place).
  • The inter-annotator agreement score of 0.81, while considered good, is at the lower end of near-perfect agreement, indicating some subjective challenges in annotation.
  • Future Work:
    • Improving event argument detection, particularly for "Capabilities" and "Purpose" roles.
  • Developing components to link event arguments to Wikidata entities.
  • Implementing event coreference and sequence relation detection.
  • Expanding the system to support additional event types.
  • Aligning and mapping extracted information to standardized cybersecurity ontologies (e.g., Unified Cybersecurity Ontology, STIX CTI schema).
  • Exporting extracted information into an integrated knowledge graph.

Summary updated

BibTeX
@inproceedings{satyapanich2020casie,
  author = {Taneeya Satyapanich and Francis Ferraro and Tim Finin},
  title = {{CASIE: Extracting Cybersecurity Event Information from Text}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6401},
}
Figure