Abstract: Timely patching is paramount to safeguard users and maintainers against dire consequences of malicious attacks. In practice, patching is prioritized following the nature of the code change that is committed in the code repository. When such a change is labeled as being security-relevant, i.e., as fixing a vulnerability, maintainers rapidly spread the change and users are notified about the need to update. Unfortunately, oftentimes, some security-relevant changes go unnoticed as they represent silent fixes of vulnerabilities. In this paper, we propose a Co-Training-based approach to catch security patches as part of an automatic monitoring service of code repositories. Leveraging different classes of features, we empirically show that such automation is feasible and can yield a precision of over 90% in identifying security patches, with an unprecedented recall of over 80%. Beyond such a benchmarking with ground truth data, we confirmed that our approach can help catch security patches that were not reported as such.
TL;DR
Security vulnerabilities are frequently fixed "silently" in code repositories without being labeled as security-relevant, leaving users exposed to attacks. This paper introduces a semi-supervised Co-Training approach that identifies these hidden patches with over 90% precision by independently analyzing commit messages and the underlying code changes.
The Problem: Silent Fixes and the Window of Vulnerability
The "window of vulnerability" is the critical time between a bug's discovery and the application of a patch. While major outbreaks like WannaCry exploited known vulnerabilities, many users remained exposed simply because they didn't realize available patches were security-critical.
A major gap exists in the software ecosystem: only about 9% of maintainers file for a formal CVE (Common Vulnerability Enumeration) ID after fixing a vulnerability, and 25% of open-source projects fix vulnerabilities completely silently. If a commit log vaguely states "fix minor bug" instead of "fix buffer overflow," system administrators are unlikely to prioritize the update. This research aims to automate the detection of these "silent fixes" at commit-time using only the information available within version control systems.
Methodology: The Power of Co-Training
The core innovation is the use of Co-Training, a semi-supervised machine learning technique that builds a robust model by leveraging two "independent views" of the same data:
- View 1: Commit Text Features: The model analyzes natural language in commit logs (e.g., "bounds check," "overflow") using a bag-of-words approach and TF-IDF.
- View 2: Code Change Features: The model examines the "diff" itself. The authors engineered specific security-sensitive features based on common vulnerability patterns, such as changes to
sizeofcalls,gotoorbreakstatements, and array index modifications.
The Co-Training Process: Starting with a small set of labeled patches, the researchers trained two separate Support Vector Machine (SVM) classifiers—one for text and one for code. These classifiers then "taught" each other by labeling a massive pool of unlabeled commits. If the "Text" classifier was highly confident a patch was security-related, it added that patch to the "Code" classifier's training set, and vice versa.
Data Collection and Labeling
The study utilized data from three major open-source C projects: the Linux Kernel, OpenSSL, and Wireshark. To train and validate the model, the researchers categorized patches into three distinct groups based on their provenance and metadata.
- Positive Data: 2,879 patches explicitly linked to CVEs.
- Negative Data: 4,411 "pure" bug fixes (linked to non-security bugs) and 1,957 code enhancement patches.
- Unlabeled Data: 166,250 patches that mentioned "bug" or "fix" but lacked an official security designation.
Key Insights
- The "Silent" Reality: In OpenSSL, a staggering 98.78% of security patches did not contain any security-related keywords in their commit logs. This confirms that text analysis alone is insufficient.
- Independence of Views: Commit logs and code changes are often "redundant but independent." A developer might write a vague log message, but the code change (e.g., adding a bounds check) clearly signals a security fix.
- Feature Engineering: Specific human-engineered features (like the frequency of
sizeoforifstatements normalized by patch size) provided higher discriminative power than raw code analysis. - Semi-Supervised Gains: By leveraging unlabeled data through Co-Training, the model's accuracy jumped from 58% (using only labeled data) to 79%.
Results and Real-World Impact
- High Performance: The approach achieved a precision of over 90% and a recall of over 80% on benchmark data, significantly outperforming previous state-of-the-art methods which typically hovered around 60%.
- Cross-Project Utility: A model trained on OpenSSL could identify security patches in the Linux kernel, suggesting that security-fixing patterns are somewhat universal across C projects.
- Real-World Validation: When the model flagged 10 "unlabeled" patches in the wild, human auditors confirmed that 55% of them were indeed likely security fixes that had gone unreported.
Limitations and Future Work
The current feature set is tailored for the C programming language. Adapting this to memory-safe languages like Java or Python would require a different set of engineered features. Future work involves expanding the approach to Java projects and investigating more complex "Boosting" algorithms to further refine classification for real-time monitoring services.
Summary updated