TL;DR
This paper introduces FVF (Fixed Vulnerability Filter), a novel framework that leverages code change history to identify and filter "Similar but Patched" (SBP) code, a common source of false alarms in vulnerability detection tools. By analyzing commit logs, FVF significantly reduces false positives from existing tools and demonstrates the limitations of current deep learning approaches in distinguishing SBP code.
What Problem Does This Paper Address?
Software development heavily relies on code reuse, which unfortunately can propagate vulnerabilities. While tools exist to find these recurring vulnerabilities, they often struggle with a specific type of code: "Similar but Patched" (SBP) code. SBP code is nearly identical to vulnerable code but has been fixed, yet it's similar enough to fool many detection tools, leading to a flood of false alarms. This forces developers to manually sift through these false positives, a time-consuming and error-prone process that hinders effective vulnerability management. Existing solutions are often language-specific or fail to handle subtle differences and code changes, leaving a critical gap in accurately identifying real vulnerabilities.
Methods
The core contribution is the Fixed Vulnerability Filter (FVF), a programming language-agnostic framework designed to post-process the output of existing vulnerability detectors. FVF operates by analyzing code change histories. When a vulnerability detector flags a piece of code as potentially vulnerable, FVF:
- Generates a Function Change Log: It uses version control system (like Git) history to record all code modifications within the detected function.
- Finds Vulnerability Patch Logs: It queries a database of known vulnerabilities to retrieve the specific code changes (patch logs) that fixed the original vulnerability.
- Matches Fix Behaviors: It compares the function's change log with the vulnerability's patch log. If the fix behaviors described in the patch log are found within the function's change history, FVF identifies it as SBP code and flags it as a false alarm.
This approach moves beyond simple code similarity by looking at how code evolved, allowing it to distinguish between truly vulnerable code and code that was once vulnerable but has since been patched.
Datasets
The paper introduces a new real-world SBP dataset containing 6,827 SBP functions collected from 1,081 open-source projects written in C, C++, and Java. This dataset was constructed using FVF and serves as a benchmark to evaluate the sensitivity of vulnerability detection tools in distinguishing real vulnerabilities from SBP code. Existing datasets like Devign and Big-Vul were used as sources for vulnerability fix commits.
Key Insights and Ideas
- The "Similar but Patched" (SBP) problem is pervasive: The authors found that 40% of the studied open-source projects contained at least one instance of SBP code, highlighting its significant impact on vulnerability detection.
- Code change history is crucial: Simple code similarity is insufficient. Analyzing how code was fixed (the patch history) is key to differentiating between actual vulnerabilities and patched code.
- Current DL approaches struggle with SBP: Even state-of-the-art deep learning models, which have shown promise in vulnerability detection, perform poorly on SBP code, often misclassifying it as vulnerable. This suggests a need for datasets and models that explicitly account for SBP.
- Patch Reversion is a common SBP scenario: A significant portion of SBP cases (52.9%) arise from patch reversions, where a fix is undone, often due to new bugs or contextual changes. FVF's ability to analyze change logs makes it adept at handling these cases.
Key Results
- FVF significantly reduces false alarms from four baseline vulnerability detection tools (ReDeBug, VUDDY, MVP, and a hash-based approach). The overall False Alarm Rate (FAR) dropped from 76.2% to 26.6%, an improvement of 49.6%.
- FVF achieved a 100% precision rate in identifying SBP code, meaning it did not incorrectly flag real vulnerabilities as SBP (no false positives).
- When tested on the newly created SBP dataset, four state-of-the-art deep learning-based vulnerability detection approaches exhibited high false alarm rates, ranging from 62.9% to 64.9%, demonstrating their inability to distinguish SBP code.
- The paper categorizes filtered SBP code into three types: Patch Reversion (52.9%), Minor Difference (36.1%), and Customized Patch (10.9%).
Limitations and Future Work
- False Negatives: While FVF has no false positives, it does have false negatives (missed SBP cases). These are primarily due to:
- Lack of vulnerability trigger point: Cases where contextual conditions are not met, requiring inter-procedural analysis, which is outside FVF's current scope.
- Trivial clone false alarms: Short, auxiliary functions or non-function code that are hard to associate with vulnerabilities, often produced by tools like ReDeBug.
- Future Work: The authors suggest extending FVF to handle inter-procedural vulnerabilities (vulnerabilities spanning multiple functions) by generating multi-function patch logs. They also highlight the need for further research into the unique characteristics of SBP scenarios and their impact on detection. The SBP dataset is intended to facilitate more realistic evaluations of vulnerability detection tools and encourage further exploration in this area.
Summary updated