Collecting Indicators of Compromise from Unstructured Text of Cybersecurity Articles using Neural-Based Sequence Labelling

Zi Long, Lianzhi Tan, Shengping Zhou, Chaoyang He, Xin Liu

TL;DR

Researchers from Tencent have developed an end-to-end neural sequence labelling model designed to automatically extract Indicators of Compromise (IOCs)—like malicious URLs, IP addresses, and file hashes—from unstructured cybersecurity articles. By combining multi-head self-attention with automatically extracted contextual features, the model achieves high performance (89.0% F1-score on English data) without requiring the hand-crafted rules or deep domain expertise that traditional systems rely on.

What Problem Does This Paper Address?

In the world of cybersecurity, "Indicators of Compromise" (IOCs) are the breadcrumbs left behind by attackers. They are forensic artifacts—MD5 hashes of malware, specific domain names used by botnets, or IP addresses of command-and-control servers. Security analysts rely on these to detect and block threats early.

However, most IOC data is buried in unstructured text: blog posts, threat intelligence reports, and technical articles. Currently, extracting this data is a nightmare. Existing automated systems usually depend on "hand-crafted features"—complex, hard-coded rules written by experts that are specific to one language or one type of threat. As cyber threats evolve at breakneck speed, these manual systems can't keep up, and human analysts are overwhelmed by the sheer volume of reports.

Methods

The authors propose a neural network architecture based on the Bi-LSTM-CRF (Bidirectional Long Short-Term Memory with Conditional Random Fields) framework, but with several key enhancements:

  1. Hybrid Embeddings: The model represents tokens using both word-level embeddings (Word2vec) and character-level embeddings (via a Bi-LSTM). This helps the model handle "out-of-vocabulary" words, which are common in technical reports (e.g., weirdly named malware files).
  2. Multi-Head Self-Attention: Borrowing from the Transformer architecture, this module allows the model to weigh the importance of different words in a sentence regardless of their distance. This helps capture global context that a standard LSTM might miss.
  3. Contextual Features: This is the "secret sauce." The researchers noticed that IOCs often appear near specific keywords like "download," "malware," or "request." They developed a method to automatically extract these "contextual keywords" from the training data and use their co-occurrence frequency as a feature.
  4. CRF Layer: The final layer ensures that the predicted labels make sense as a sequence (e.g., an "Inside-IOC" label shouldn't follow an "Outside-IOC" label without a "Begin-IOC" label first).

Datasets

The study utilized two primary datasets:

  • English Dataset: 687 cybersecurity articles (Advanced Persistent Threat reports) published between 2008 and 2018. 370 of these were manually annotated for training and testing.
  • Chinese Dataset: 5,427 articles crawled from 35 different cybersecurity blogs. 607 were manually annotated.
  • Labels: The researchers used 11 distinct labels (e.g., attacker, attack method, file hash, CVE vulnerability) using the BIO (Begin, Inside, Outside) tagging scheme.

Key Insights and Ideas

  • Context is King: The most significant insight is that spelling patterns (like a regex for an IP address) aren't enough. Many strings look like IOCs but are benign. By incorporating "contextual features," the model mimics a human analyst who looks for surrounding words like "malicious" or "dropped" to confirm a suspicion.
  • Attention Bridges the Gap: Cybersecurity reports often contain long lists of IOCs with very little surrounding text. The multi-head attention mechanism is particularly effective here because it can relate a token at the end of a long list back to the descriptive verb at the beginning of the paragraph.
  • Bilingual Synergy: The researchers found that training on a combined English and Chinese dataset actually improved performance on both, particularly for the Chinese test set (boosting the F1-score from 81.8% to 84.4%). This suggests that the contextual patterns of how threats are described are somewhat universal across languages.

Key Results

The proposed model consistently outperformed several state-of-the-art baselines:

  • English Performance: Achieved an 89.0% F1-score, significantly higher than standard NER (Named Entity Recognition) models like those proposed by Lample et al. (78.9%) or Huang et al. (51.6%).
  • Chinese Performance: Achieved an 81.8% F1-score (which rose to 84.4% with bilingual training).
  • Precision vs. Recall: On English data, the model reached a high precision of 93.1%, meaning when it identifies something as an IOC, it is very likely correct—a crucial metric for reducing "alert fatigue" in security operations.

Limitations and Future Work

While the model is highly effective, the authors note a few areas for improvement:

  • Contextual Embeddings: The paper was written just as large-scale pretrained language models (like BERT) were becoming dominant. The authors suggest that integrating deeper contextual embeddings from bidirectional language models could further boost performance.
  • Dataset Size: While the model performs well on small datasets, the manual annotation of cybersecurity text remains a bottleneck. Future work might explore semi-supervised or unsupervised learning to leverage the thousands of unlabelled articles available.
  • Complex IOCs: Some IOCs are more "behavioral" than string-based; the current model focuses on discrete tokens, which might miss more complex, multi-step attack patterns described across multiple sentences.

Summary updated

Abstract

Indicators of Compromise (IOCs) are artifacts observed on a network or in an operating system that can be utilized to indicate a computer intrusion and detect cyber-attacks in an early stage. Thus, they exert an important role in the field of cybersecurity. However, state-of-the-art IOCs detection systems rely heavily on hand-crafted features with expert knowledge of cybersecurity, and require large-scale manually annotated corpora to train an IOC classifier. In this paper, we propose using an end-to-end neural-based sequence labelling model to identify IOCs automatically from cybersecurity articles without expert knowledge of cybersecurity. By using a multi-head self-attention module and contextual features, we find that the proposed model is capable of gathering contextual information from texts of cybersecurity articles and performs better in the task of IOC identification. Experiments show that the proposed model outperforms other sequence labelling models, achieving the average F1-score of 89.0% on English cybersecurity article test set, and approximately the average F1-score of 81.8% on Chinese test set.
BibTeX
@inproceedings{long2019collecting,
  author = {Zi Long and Lianzhi Tan and Shengping Zhou and Chaoyang He and Xin Liu},
  title = {{Collecting Indicators of Compromise from Unstructured Text of Cybersecurity Articles using Neural-Based Sequence Labelling}},
  booktitle = {IJCNN},
  year = {2019},
  url = {https://openreview.net/forum?id=D5c2QSkKa9k},
}

Loading related content...

Figure