DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection

Yizheng Chen, Zhoujie Ding, Xinyun Chen, David A. Wagner

Abstract

We propose and release a new vulnerable source code dataset, DiverseVul. We curate the dataset by crawling security issue websites, extracting vulnerability-fixing commits and source codes from the corresponding projects. Our new dataset contains 18,945 vulnerable functions spanning 150 CWEs and 330,492 non-vulnerable functions extracted from 7,514 commits. Our dataset covers 295 more projects than all previous datasets combined. Combining our new dataset with previous datasets, we present an analysis of the challenges and promising research directions of using deep learning for detecting software vulnerabilities. We study 11 model architectures belonging to 4 families. Our results show that deep learning is still not ready for vulnerability detection, due to high false positive rate, low F1 score, and difficulty of detecting hard CWEs. In particular, we demonstrate an important generalization challenge for the deployment of deep learning-based models. We show that increasing the volume of training data may not further improve the performance of deep learning models for vulnerability detection, but might be useful to improve the generalization ability to unseen projects. We also identify hopeful future research directions. We demonstrate that large language models (LLMs) are a promising research direction for ML-based vulnerability detection, outperforming Graph Neural Networks (GNNs) with code-structure features in our experiments. Moreover, developing source code specific pre-training objectives is a promising research direction to improve the vulnerability detection performance.

Introduction

Detecting software vulnerabilities is a critical task in software engineering, yet it remains a high-stakes game of cat-and-mouse. While deep learning (DL) has revolutionized fields like computer vision and natural language processing, its application to source code analysis has been hampered by datasets that are often small, low-diversity, or synthetic. Existing benchmarks frequently lack the "real-world" complexity of production code or focus on a narrow set of Common Weakness Enumerations (CWEs). Furthermore, a fundamental debate persists: should models rely on explicit code structures, such as Abstract Syntax Trees (ASTs) via Graph Neural Networks (GNNs), or can text-based Large Language Models (LLMs) implicitly learn these patterns?

This research introduces DiverseVul, a large-scale, diverse dataset for C/C++ vulnerability detection designed to address these gaps and provide a rigorous benchmark for modern DL architectures.

The DiverseVul Dataset

DiverseVul was curated by crawling security issue trackers, including Snyk and Red Hat Bugzilla, to identify vulnerability-fixing commits. The researchers employed a standard "before-and-after" labeling heuristic: the version of a function immediately preceding a security fix is labeled vulnerable, while the version following the fix is labeled non-vulnerable.

Scale and Diversity

DiverseVul represents a significant expansion over existing resources:

  • Volume: 18,945 vulnerable and 330,492 non-vulnerable functions.
  • Breadth: Coverage of 150 distinct CWEs across 797 projects.
  • Comparison: The dataset includes 295 more projects than all previous major datasets (Devign, ReVeal, BigVul, CrossVul, and CVEFixes) combined.

Benchmarking Architectures

The study evaluated 11 models across four distinct families to determine the current state of the art:

  1. GNNs: Specifically ReVeal, which utilizes Code Property Graphs to capture structural relationships.
  2. RoBERTa-based: Encoder-only models such as CodeBERT.
  3. GPT-based: Decoder-only models such as PolyCoder.
  4. T5-based: Encoder-decoder models including CodeT5 and NatGen.

The evaluation focused on two primary dimensions: how performance scales with data volume and how effectively models generalize to "unseen" projects not included in the training set.

Key Insights and Results

The Shift from GNNs to LLMs

Historically, GNNs were favored for their ability to model code structure explicitly. However, the study found that as training data volume increases, LLMs—particularly T5-based variants—significantly outperform GNNs. LLMs appear more capable of capturing the nuanced patterns present in large-scale, diverse datasets. Notably, the NatGen model achieved the highest F1 score of 47.15% on the full dataset.

The Power of Code-Specific Pre-training

Model size is not the sole determinant of success. The researchers found that CodeT5 Small (60M parameters) often outperformed larger models like RoBERTa (125M parameters). This is attributed to code-specific pre-training objectives (e.g., predicting variable names or code generation) which provide a better foundational understanding of programming logic than general natural language pre-training.

The Generalization Wall

Perhaps the most significant finding is the "Generalization Gap." When models are tested on a random split of functions—where the same project may appear in both training and testing—they perform reasonably well (F1 ~49%). However, when tested on unseen projects, performance craters to approximately 9.4%. This suggests that current models are largely overfitting to project-specific coding styles and idioms rather than learning the underlying characteristics of vulnerabilities.

Data Quality and Noise

Manual audits of the "fix-commit" labeling method revealed a significant noise problem. The heuristic is only approximately 60% accurate, as many security commits include irrelevant changes like whitespace adjustments or refactoring. This noise confuses models and limits their ability to learn clean signals.

Conclusion and Future Directions

The results indicate that deep learning is not yet ready for "plug-and-play" deployment in vulnerability detection. Even the best models maintain a False Positive Rate (FPR) of roughly 3.5%, which would overwhelm developers with hundreds of false alarms in large codebases.

To move forward, the researchers suggest:

  1. Cross-Project Evaluation: Future benchmarks must prioritize testing on unseen projects to ensure genuine learning.
  2. Cleaner Datasets: There is a critical need for better filtering of non-security changes in commit-based datasets.
  3. Architectural Innovation: Simply increasing data volume for current architectures may be reaching diminishing returns; new pre-training objectives specifically designed for source code are likely required to break through the generalization wall.

Summary updated

Abstract

We propose and release a new vulnerable source code dataset. We curate the dataset by crawling security issue websites, extracting vulnerability-fixing commits and source codes from the corresponding projects. Our new dataset contains 18,945 vulnerable functions spanning 150 CWEs and 330,492 non-vulnerable functions extracted from 7,514 commits. Our dataset covers 295 more projects than all previous datasets combined. Combining our new dataset with previous datasets, we present an analysis of the challenges and promising research directions of using deep learning for detecting software vulnerabilities. We study 11 model architectures belonging to 4 families. Our results show that deep learning is still not ready for vulnerability detection, due to high false positive rate, low F1 score, and difficulty of detecting hard CWEs. In particular, we demonstrate an important generalization challenge for the deployment of deep learning-based models. We show that increasing the volume of training data may not further improve the performance of deep learning models for vulnerability detection, but might be useful to improve the generalization ability to unseen projects. We also identify hopeful future research directions. We demonstrate that large language models (LLMs) are a promising research direction for ML-based vulnerability detection, outperforming Graph Neural Networks (GNNs) with code-structure features in our experiments. Moreover, developing source code specific pre-training objectives is a promising research direction to improve the vulnerability detection performance.
BibTeX
@inproceedings{chen2023diversevul,
  author = {Yizheng Chen and Zhoujie Ding and Xinyun Chen and David A. Wagner},
  title = {{DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection}},
  booktitle = {RAID},
  year = {2023},
  doi = {10.1145/3607199.3607242},
}
Figure