Abstract
Automated software vulnerability detection using Machine Learning (ML) and Natural Language Processing (NLP) has gained significant traction. However, the impact of specific design choices—such as tokenization methods, embedding techniques, and architectural metrics—remains under-explored. This paper presents a systematic evaluation of these factors through 408 experiments across 17 real-world projects. Our findings reveal that a simple combination of Bag-of-Words (BoW) embeddings and a Random Forest classifier provides a robust baseline, often outperforming complex deep learning models.
Introduction
As software complexity scales, manual security audits become increasingly impractical. Researchers have turned to ML and NLP to "read" source code and identify potential vulnerabilities. Despite progress, there is no consensus on the optimal "ingredients" for these models. Key questions remain: Do code comments provide useful signals? Are advanced embeddings like Word2Vec superior to simple frequency-based methods? Do high-level architectural metrics offer better predictive power than raw code text? This study aims to establish a baseline for the state-of-the-art and address the challenges of transfer learning in vulnerability detection.
Methodology
The researchers conducted a comprehensive study involving 408 experiments to test 10 hypotheses. The experimental framework, illustrated in Figure 1, decomposes the learning process into four primary components: tokenization, embeddings, architectural metrics, and model selection.
The study utilized three diverse datasets: the OWASP Benchmark (2,740 cases), the Juliet Test Suite for Java (514 files), and a collection of 15 real-world Android applications labeled via static analysis. The feature engineering pipeline, shown in Figure 2, transforms raw source code into numerical vectors through tokenization and embedding before feeding them into classifiers like Random Forest or Residual Neural Networks (ResNet).
Key Insights and Results
The Power of Simplicity
One of the most striking findings is the "simplicity paradox." The combination of Bag-of-Words (BoW) and Random Forest (RF) consistently outperformed more sophisticated deep learning architectures like ResNet. Across all experiments, 95% of the learning metrics (precision, recall, F1-score) were above 0.77. On average, the BoW+RF baseline increased detection accuracy by approximately 4% and achieved precision and recall scores 8% higher than SVM or ResNet in most scenarios.
Noise vs. Signal
The experiments showed that filtering out code comments and special symbols had no statistically significant impact on model performance. This suggests that vulnerability signatures are primarily embedded in the code's logic and structure rather than developer annotations. Furthermore, token-based embeddings proved to be the dominant predictive factor; adding architectural metrics did not yield significant improvements.
The Transferability Gap
A major challenge identified is the "transferability wall." Models trained on synthetic datasets like Juliet performed poorly when applied to real-world Android applications. This indicates a significant domain gap, where vulnerability patterns do not easily translate across different project types or environments.
Conclusion and Future Directions
This research underscores the importance of establishing strong, simple baselines in the field of ML-based vulnerability detection. While token-based models are effective, the difficulty in cross-project transfer learning remains a hurdle. Future work should focus on integrating these models into continuous DevOps pipelines, allowing them to adapt to the specific nuances of target domains. Additionally, exploring graph-based models may help capture deeper semantic relationships currently missed by "flat" token-based approaches.
Summary updated