Abstract
As software development projects increasingly rely on open-source software, users face the risk of security vulnerabilities from third-party libraries. To address label and character noise in code changes, we present VF-Detector to automatically identify bug-fix commits in actual noisy development environments. VF-Detector consists of three components: Data Pre-processing (DP), Vulnerability Confidence Computation (VCC), and Confidence Learning Denoising (CLD). The DP component is responsible for preprocessing code change data. The VCC component calculates code change confidence values for each bug-fix by extracting features at various granularity levels. The CLD component removes noise and enhances model robustness by pruning noisy data with confidence values and performing effort-aware adjustments. Experimental results demonstrate VF-Detector's superiority over state-of-the-art methods in EffortCost@L and Popt@L metrics on Java and Python datasets. The improvements were 6.5% and 5% for Java, and 23.4% and 17.8% for Python.
Introduction
Modern software development is heavily dependent on third-party open-source libraries. While this accelerates development, it also exposes projects to security vulnerabilities. Identifying the specific code changes (commits) that fix these vulnerabilities is crucial for security auditing and patch management. However, automated detection tools face two significant challenges: Label Noise, where training data contains incorrectly labeled commits, and Character Noise, where security fixes are "entangled" with irrelevant changes like refactoring or documentation updates.
Existing deep learning models often struggle with these messy, imbalanced datasets. VF-Detector addresses these issues by leveraging multi-granularity feature extraction and a robust denoising framework to improve detection accuracy and reduce the manual effort required for verification.
The VF-Detector Framework
VF-Detector is structured into three primary stages designed to handle the complexities of real-world code changes.
1. Data Pre-processing (DP)
To capture the full context of a change, the DP component decomposes every code commit into four distinct granularities: Line, Hunk, File, and Commit. This multi-level approach ensures that the model can identify vulnerability patterns whether they are localized in a single line or spread across multiple files.
2. Vulnerability Confidence Computation (VCC)
This component serves as the feature extraction engine. It utilizes CodeBERT to generate initial embeddings of the code. Recognizing that different granularities require different analytical approaches, VCC employs tailored neural architectures:
- BiLSTMs for sequential dependencies at the Line level.
- CNNs for capturing local logic within code Hunks.
- Fully Connected Networks (FCNs) for high-level relationships at the File and Commit levels.
3. Confidence Learning Denoising (CLD)
The CLD component is the core innovation for handling label noise. Instead of assuming all training labels are correct, it estimates the probability of a label's accuracy. Using a "Confident Learning" framework, it prunes the most likely mislabeled samples from the training set. Furthermore, it applies an Effort-Aware Adjustment, which uses a logarithmic function to prioritize smaller, more easily inspectable commits, aligning the model's output with the practical needs of human reviewers.
Experimental Evaluation
The researchers evaluated VF-Detector using extensive datasets for Java and Python:
- Java: 1,436 bug-fix commits and 474,555 non-bug-fix commits across 150 projects.
- Python: 885 bug-fix commits and 357,696 non-bug-fix commits across 106 projects.
To manage the extreme class imbalance, random undersampling was applied to the training set. The model's performance was measured using EffortCost@L (the percentage of fixes found within a specific inspection budget) and Popt@L (the normalized area under the cost-effort curve).
Key Results and Discussion
VF-Detector significantly outperformed state-of-the-art baselines, including Midas, VulFixMiner, and DeepJIT.
As shown in the results table, VF-Detector achieved the highest scores in nearly all effort-based metrics. In the Java dataset, it improved EffortCost@L by 6.5% and Popt@L by 5%. The results were even more pronounced in the Python dataset, with improvements of 23.4% and 17.8%, respectively. Notably, at a 20% inspection budget, VF-Detector identified over 90% of vulnerability-fixing commits in Java.
The Impact of Denoising
A critical insight from the study is that "less is more." By pruning approximately 10% of the most uncertain (noisy) samples, the model achieved better generalization. This confirms that cleaning the training data is more effective than simply increasing the volume of noisy data.
Conclusion
VF-Detector provides a robust solution for identifying security patches in the noisy environment of open-source development. By combining multi-granularity feature extraction with confidence-based denoising and effort-aware adjustments, it provides a tool that is not only more accurate but also more practical for real-world security auditing. Future work will focus on expanding the framework to other programming languages and refining the denoising process for positive samples.
Summary updated