Explaining the Contributing Factors for Vulnerability Detection in Machine Learning

Esma Mouine, Yan Liu, Lu Xiao, Rick Kazman, Xiao WangarXiv 2024 · DOI · arXiv

Abstract

Automated software vulnerability detection using Machine Learning (ML) and Natural Language Processing (NLP) has gained significant traction. However, the impact of specific design choices—such as tokenization methods, embedding techniques, and architectural metrics—remains under-explored. This paper presents a systematic evaluation of these factors through 408 experiments across 17 real-world projects. Our findings reveal that a simple combination of Bag-of-Words (BoW) embeddings and a Random Forest classifier provides a robust baseline, often outperforming complex deep learning models.

Introduction

As software complexity scales, manual security audits become increasingly impractical. Researchers have turned to ML and NLP to "read" source code and identify potential vulnerabilities. Despite progress, there is no consensus on the optimal "ingredients" for these models. Key questions remain: Do code comments provide useful signals? Are advanced embeddings like Word2Vec superior to simple frequency-based methods? Do high-level architectural metrics offer better predictive power than raw code text? This study aims to establish a baseline for the state-of-the-art and address the challenges of transfer learning in vulnerability detection.

Methodology

The researchers conducted a comprehensive study involving 408 experiments to test 10 hypotheses. The experimental framework, illustrated in Figure 1, decomposes the learning process into four primary components: tokenization, embeddings, architectural metrics, and model selection.

Overview of the experimental methodology and research questions.

The study utilized three diverse datasets: the OWASP Benchmark (2,740 cases), the Juliet Test Suite for Java (514 files), and a collection of 15 real-world Android applications labeled via static analysis. The feature engineering pipeline, shown in Figure 2, transforms raw source code into numerical vectors through tokenization and embedding before feeding them into classifiers like Random Forest or Residual Neural Networks (ResNet).

The feature engineering pipeline and the Residual Block architecture used for classification.

Key Insights and Results

The Power of Simplicity

One of the most striking findings is the "simplicity paradox." The combination of Bag-of-Words (BoW) and Random Forest (RF) consistently outperformed more sophisticated deep learning architectures like ResNet. Across all experiments, 95% of the learning metrics (precision, recall, F1-score) were above 0.77. On average, the BoW+RF baseline increased detection accuracy by approximately 4% and achieved precision and recall scores 8% higher than SVM or ResNet in most scenarios.

Noise vs. Signal

The experiments showed that filtering out code comments and special symbols had no statistically significant impact on model performance. This suggests that vulnerability signatures are primarily embedded in the code's logic and structure rather than developer annotations. Furthermore, token-based embeddings proved to be the dominant predictive factor; adding architectural metrics did not yield significant improvements.

The Transferability Gap

A major challenge identified is the "transferability wall." Models trained on synthetic datasets like Juliet performed poorly when applied to real-world Android applications. This indicates a significant domain gap, where vulnerability patterns do not easily translate across different project types or environments.

Conclusion and Future Directions

This research underscores the importance of establishing strong, simple baselines in the field of ML-based vulnerability detection. While token-based models are effective, the difficulty in cross-project transfer learning remains a hurdle. Future work should focus on integrating these models into continuous DevOps pipelines, allowing them to adapt to the specific nuances of target domains. Additionally, exploring graph-based models may help capture deeper semantic relationships currently missed by "flat" token-based approaches.

Summary updated

Abstract

There is an increasing trend to mine vulnerabilities from software repositories and use machine learning techniques to automatically detect software vulnerabilities. A fundamental but unresolved research question is: how do different factors in the mining and learning process impact the accuracy of identifying vulnerabilities in software projects of varying characteristics? Substantial research has been dedicated in this area, including source code static analysis, software repository mining, and NLP-based machine learning. However, practitioners lack experience regarding the key factors for building a baseline model of the state-of-the-art. In addition, there lacks of experience regarding the transferability of the vulnerability signatures from project to project. This study investigates how the combination of different vulnerability features and three representative machine learning models impact the accuracy of vulnerability detection in 17 real-world projects. We examine two types of vulnerability representations: 1) code features extracted through NLP with varying tokenization strategies and three different embedding techniques (bag-of-words, word2vec, and fastText) and 2) a set of eight architectural metrics that capture the abstract design of the software systems. The three machine learning algorithms include a random forest model, a support vector machines model, and a residual neural network model. The analysis shows a recommended baseline model with signatures extracted through bag-of-words embedding, combined with the random forest, consistently increases the detection accuracy by about 4% compared to other combinations in all 17 projects. Furthermore, we observe the limitation of transferring vulnerability signatures across domains based on our experiments.
BibTeX
@misc{mouine2024explaining,
  author = {Esma Mouine and Yan Liu and Lu Xiao and Rick Kazman and Xiao Wang},
  title = {{Explaining the Contributing Factors for Vulnerability Detection in Machine Learning}},
  year = {2024},
  eprint = {2406.03577},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.03577},
}
Figure