TL;DR
This paper introduces CTI-BERT, a new language model specifically trained for cybersecurity. By creating a specialized vocabulary and training from scratch on a large corpus of cybersecurity texts, CTI-BERT significantly outperforms general-purpose models on various cybersecurity tasks, paving the way for more effective automated threat intelligence analysis.
What Problem Does This Paper Address?
The cybersecurity landscape is constantly evolving with an overwhelming volume of new threats, vulnerabilities, and attack techniques reported daily. While cybersecurity experts diligently publish detailed reports, manually sifting through this information to extract actionable intelligence is a monumental task. Existing general-purpose language models (like standard BERT) struggle with the unique terminology and writing styles prevalent in cybersecurity, leading to suboptimal performance. Domain-specific language models are crucial, but previous attempts have either relied on continued pre-training (which can limit vocabulary expansion) or lacked comprehensive evaluation across a wide range of cybersecurity tasks. This paper aims to bridge this gap by developing and rigorously evaluating a new, domain-specific language model trained from scratch for cybersecurity.
Methods
The core of this work is the development of CTI-BERT, a BERT-based language model. The authors chose to train CTI-BERT from scratch, which involves creating a new vocabulary tailored to the cybersecurity domain. This is a key differentiator from "continual pre-training" approaches, which build upon existing general-domain models and may struggle with domain-specific terms that are out-of-vocabulary.
The training process involved:
- Custom Tokenizer: A WordPiece tokenizer was trained on the curated cybersecurity corpus to generate a domain-specific vocabulary of 50,000 tokens.
- Masked Language Modeling (MLM): CTI-BERT was trained using the MLM objective, a standard self-supervised learning task where the model learns to predict masked tokens in a sequence.
- Extensive Fine-tuning and Evaluation: The pre-trained CTI-BERT model was then fine-tuned on a variety of downstream cybersecurity tasks, including:
- Masked Word Prediction: To assess the model's understanding of domain-specific terms.
- Sentence/Document Classification: Tasks like ATT&CK technique classification and IoT app description classification.
- Token Classification: Named Entity Recognition (NER) for coarse-grained and fine-grained security entities, and token type classification.
- Malware Sentence Detection and Attribute Classification.
Datasets
The authors curated a comprehensive cybersecurity corpus from various reputable sources, including:
- Attack Descriptions: From MITRE ATT&CK and CAPEC, providing details on known cyber-attack techniques.
- Security Textbooks: Two online textbooks for CISSP certification.
- Academic Papers: From the USENIX Security Symposium (1990-2021).
- Security Wiki: Wikipedia pages related to "Computer Security."
- Threat Reports: News articles and white papers on cyber-campaigns, malware, and vulnerabilities.
- Vulnerability Data: From CVE and CWE.
For evaluation, they utilized existing datasets and introduced new benchmarks for specific tasks, such as:
- TRAM: A dataset for ATT&CK technique classification.
- IoTSpotter: For IoT app description classification.
- MalwareTextDB: For malware sentence detection and attribute classification.
- STIX-based datasets: For NER tasks.
Key Insights and Ideas
- Training from Scratch Matters: The paper strongly suggests that training a language model from scratch with a domain-specific vocabulary (like CTI-BERT) is critical for achieving superior performance in specialized domains like cybersecurity, outperforming models that only continue pre-training.
- Vocabulary is King: The custom vocabulary generated for CTI-BERT allows it to better understand and predict domain-specific terms, as demonstrated by its strong performance in masked word prediction, even identifying specific malware families like RAT.
- Comprehensive Evaluation is Crucial: The authors' extensive evaluation across a wide array of sentence- and token-level tasks provides strong evidence for CTI-BERT's effectiveness, highlighting its superiority over general-domain and even other security-specific models.
- Domain Models Improve Specific Tasks: CTI-BERT consistently shows significant improvements, particularly in tasks requiring nuanced understanding of cybersecurity jargon, such as ATT&CK technique classification and fine-grained NER.
Key Results
CTI-BERT consistently outperformed baseline models, including bert-base-uncased, SecBERT, roberta-base, SecROBERTa, and SecureBERT, across most evaluated tasks.
- ATT&CK Technique Classification: CTI-BERT achieved the highest Micro-F1 score (85.18) and Macro-F1 score (69.26), outperforming other models by a "large margin."
- IoT App Classification: CTI-BERT achieved a high Micro-F1 score of 96.40, slightly outperforming
bert-base-uncasedandroberta-base. - Malware Sentence Detection: CTI-BERT and
SecROBERTaperformed well on this task. - NER Tasks: CTI-BERT showed strong performance, particularly in fine-grained NER, where it outperformed other models by a "large margin."
- Masked Word Prediction: CTI-BERT demonstrated excellent understanding, often predicting the correct word or a relevant synonym, while other models struggled.
Limitations and Future Work
The authors acknowledge several limitations:
- English-Only: The model is currently pretrained using only English data. Future work could involve adding support for multiple languages.
- Dataset Size: The benchmark datasets used for evaluation, while comprehensive, are relatively small. The authors suggest that further evaluations with larger datasets are needed to confirm findings.
Future work directions include:
- Collecting more documents to further improve the model.
- Training other language model architectures to support different security applications.
Summary updated