Similar but Patched Code Considered Harmful -- The Impact of Similar but Patched Code on Recurring Vulnerability Detection and How to Remove Them

Zheng‐Hua Tan, Jiayuan Zhou, Xing Hu, Shengyi Pan, Kui Liu, Xin Xia

TL;DR

This paper introduces FVF (Fixed Vulnerability Filter), a novel framework that leverages code change history to identify and filter "Similar but Patched" (SBP) code, a common source of false alarms in vulnerability detection tools. By analyzing commit logs, FVF significantly reduces false positives from existing tools and demonstrates the limitations of current deep learning approaches in distinguishing SBP code.

What Problem Does This Paper Address?

Software development heavily relies on code reuse, which unfortunately can propagate vulnerabilities. While tools exist to find these recurring vulnerabilities, they often struggle with a specific type of code: "Similar but Patched" (SBP) code. SBP code is nearly identical to vulnerable code but has been fixed, yet it's similar enough to fool many detection tools, leading to a flood of false alarms. This forces developers to manually sift through these false positives, a time-consuming and error-prone process that hinders effective vulnerability management. Existing solutions are often language-specific or fail to handle subtle differences and code changes, leaving a critical gap in accurately identifying real vulnerabilities.

Methods

The core contribution is the Fixed Vulnerability Filter (FVF), a programming language-agnostic framework designed to post-process the output of existing vulnerability detectors. FVF operates by analyzing code change histories. When a vulnerability detector flags a piece of code as potentially vulnerable, FVF:

  1. Generates a Function Change Log: It uses version control system (like Git) history to record all code modifications within the detected function.
  2. Finds Vulnerability Patch Logs: It queries a database of known vulnerabilities to retrieve the specific code changes (patch logs) that fixed the original vulnerability.
  3. Matches Fix Behaviors: It compares the function's change log with the vulnerability's patch log. If the fix behaviors described in the patch log are found within the function's change history, FVF identifies it as SBP code and flags it as a false alarm.

This approach moves beyond simple code similarity by looking at how code evolved, allowing it to distinguish between truly vulnerable code and code that was once vulnerable but has since been patched.

Datasets

The paper introduces a new real-world SBP dataset containing 6,827 SBP functions collected from 1,081 open-source projects written in C, C++, and Java. This dataset was constructed using FVF and serves as a benchmark to evaluate the sensitivity of vulnerability detection tools in distinguishing real vulnerabilities from SBP code. Existing datasets like Devign and Big-Vul were used as sources for vulnerability fix commits.

Key Insights and Ideas

  • The "Similar but Patched" (SBP) problem is pervasive: The authors found that 40% of the studied open-source projects contained at least one instance of SBP code, highlighting its significant impact on vulnerability detection.
  • Code change history is crucial: Simple code similarity is insufficient. Analyzing how code was fixed (the patch history) is key to differentiating between actual vulnerabilities and patched code.
  • Current DL approaches struggle with SBP: Even state-of-the-art deep learning models, which have shown promise in vulnerability detection, perform poorly on SBP code, often misclassifying it as vulnerable. This suggests a need for datasets and models that explicitly account for SBP.
  • Patch Reversion is a common SBP scenario: A significant portion of SBP cases (52.9%) arise from patch reversions, where a fix is undone, often due to new bugs or contextual changes. FVF's ability to analyze change logs makes it adept at handling these cases.

Key Results

  • FVF significantly reduces false alarms from four baseline vulnerability detection tools (ReDeBug, VUDDY, MVP, and a hash-based approach). The overall False Alarm Rate (FAR) dropped from 76.2% to 26.6%, an improvement of 49.6%.
  • FVF achieved a 100% precision rate in identifying SBP code, meaning it did not incorrectly flag real vulnerabilities as SBP (no false positives).
  • When tested on the newly created SBP dataset, four state-of-the-art deep learning-based vulnerability detection approaches exhibited high false alarm rates, ranging from 62.9% to 64.9%, demonstrating their inability to distinguish SBP code.
  • The paper categorizes filtered SBP code into three types: Patch Reversion (52.9%), Minor Difference (36.1%), and Customized Patch (10.9%).

Limitations and Future Work

  • False Negatives: While FVF has no false positives, it does have false negatives (missed SBP cases). These are primarily due to:
  • Lack of vulnerability trigger point: Cases where contextual conditions are not met, requiring inter-procedural analysis, which is outside FVF's current scope.
  • Trivial clone false alarms: Short, auxiliary functions or non-function code that are hard to associate with vulnerabilities, often produced by tools like ReDeBug.
  • Future Work: The authors suggest extending FVF to handle inter-procedural vulnerabilities (vulnerabilities spanning multiple functions) by generating multi-function patch logs. They also highlight the need for further research into the unique characteristics of SBP scenarios and their impact on detection. The SBP dataset is intended to facilitate more realistic evaluations of vulnerability detection tools and encourage further exploration in this area.

Summary updated

Abstract

Identifying recurring vulnerabilities is crucial for ensuring software security. Clone-based techniques, while widely used, often generate many false alarms due to the existence of similar but patched (SBP) code, which is similar to vulnerable code but is not vulnerable due to having been patched. Although the SBP code poses a great challenge to the effectiveness of existing approaches, it has not yet been well explored. In this paper, we propose a programming language agnostic framework, Fixed Vulnerability Filter (FVF), to identify and filter such SBP instances in vulnerability detection. Different from existing studies that leverage function signatures, our approach analyzes code change histories to precisely pinpoint SBPs and consequently reduce false alarms. Evaluation under practical scenarios confirms the effectiveness and precision of our approach. Remarkably, FVF identifies and filters 65.1% of false alarms from four vulnerability detection tools (i.e., ReDeBug, VUDDY, MVP, and an elementary hash-based approach) without yielding false positives. We further apply FVF to 1,081 real-world software projects and construct a real-world SBP dataset containing 6,827 SBP functions. Due to the SBP nature, the dataset can act as a strict benchmark to test the sensitivity of the vulnerability detection approach in distinguishing real vulnerabilities and SBPs. Using this dataset, we demonstrate the ineffectiveness of four state-of-the-art deep learning-based vulnerability detection approaches. Our dataset can help developers make a more realistic evaluation of vulnerability detection approaches and also paves the way for further exploration of real-world SBP scenarios.
BibTeX
@misc{tan2024similar,
  author = {Zheng‐Hua Tan and Jiayuan Zhou and Xing Hu and Shengyi Pan and Kui Liu and Xin Xia},
  title = {{Similar but Patched Code Considered Harmful -- The Impact of Similar but Patched Code on Recurring Vulnerability Detection and How to Remove Them}},
  year = {2024},
  eprint = {2412.20740},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.20740},
}
Figure