NAVRepair: Node-type Aware C/C++ Code Vulnerability Repair

Ruoke Wang, Zongjie Li, Chaozheng Wang, Yang Xiao, Cuiyun GaoarXiv 2024 · arXiv · OpenReview

Abstract

The rapid advancement of deep learning has led to the development of Large Language Models (LLMs). In the field of vulnerability repair, previous research has leveraged rule-based fixing, pre-trained models, and LLM's prompt engineering. However, existing approaches have limitations in terms of the integration of code structure with error types. Besides, due to certain features of C/C++ language, vulnerability repair in C/C++ proves to be exceptionally challenging. To address these challenges, we propose NAVRepair, a novel framework that combines the node-type information extracted from Abstract Syntax Trees (ASTs) with error types, specifically targeting C/C++ vulnerabilities. Specifically, our approach employs type analysis to localize the minimum edit node (MEN) and customizes context information collection based on different error types. In the offline stage, NAVRepair parses code patches to locate MENs and designs rules to extract relevant contextual information for each MEN type. In the online repairing stage, it analyzes the suspicious code, combines it with vulnerability type templates derived from the Common Weakness Enumeration (CWE), and generates targeted repair prompts. We evaluate NAVRepair on multiple popular LLMs and demonstrate its effectiveness in improving the performance of code vulnerability repair. Notably, our framework is independent of any specific LLMs and can quickly adapt to new vulnerability types. Extensive experiments validate that NAVRepair achieves excellent results in assisting LLMs to accurately detect and fix C/C++ vulnerabilities. We achieve a 26% higher accuracy compared to an existing LLM-based C/C++ vulnerability repair method. We believe our node type-aware approach has promising application prospects for enhancing real-world C/C++ code security.

Introduction

Automated Program Repair (APR) has traditionally relied on two methods: rule-based systems, which are rigid and require expert manual labor, and deep learning models, which often lack generalizability. Recently, Large Language Models (LLMs) have shown great promise, but they typically treat code as a simple sequence of text (sequence-to-sequence).

This approach creates a "syntax gap": LLMs often fail to capture the specific structural patterns associated with different vulnerability types, such as a missing boundary check in a C++ array. Without explicit guidance on the code's syntax tree, LLMs may suggest "plausible" but incorrect fixes that do not resolve the underlying security flaw. NAVRepair addresses this by bridging the gap between deep static analysis and the generative power of LLMs.

The NAVRepair Framework

The core innovation of NAVRepair is the use of the Minimum Edit Node (MEN) and a three-stage pipeline designed to provide structural context rather than just raw text.

Comparison of vulnerability repair approaches and the NAVRepair pipeline.
  1. Rule Design (Offline): The authors analyzed 547 vulnerable/fixed code pairs to identify the smallest unit of structural change required for a fix—the MEN. These are categorized into six common types such as if_statement, assignment_expression, and call_expression.
  2. Prompt Generation (Online): When a vulnerability is detected, NAVRepair identifies the MEN type of the suspicious code and applies pre-defined rules to extract "repair-relevant cues" (e.g., nearby variables missing from a conditional check). It then regenerates the code snippet to remove irrelevant blocks, keeping only the logic essential for the fix. This is combined with Common Weakness Enumeration (CWE) descriptions to provide the LLM with both the "what" (bug type) and the "how" (structural hint).
  3. Patch Generation: This enriched prompt is fed into a conversational LLM (such as GPT-4 or Gemini) to produce the final code patch.

Evaluation and Results

NAVRepair was evaluated using two primary datasets: CVEFixes, a widely used dataset of real-world C/C++ vulnerabilities, and PatchDB, which provided 300 "fresh" vulnerabilities from 2024 to prevent data leakage from the LLMs' training sets.

The framework demonstrated significant performance gains across five different LLMs (ChatGPT-3.5, ChatGPT-4, Gemini 1.5 Pro, DeepSeek Coder, and Magicoder):

  • Accuracy Boost: NAVRepair achieved a 26% higher accuracy compared to existing LLM-based C/C++ vulnerability repair methods.
  • Relative Gains: Compared to basic prompting ("Sole LLM"), NAVRepair increased successful repairs by 120% for GPT-4 and 311% for Magicoder.
  • State-of-the-Art: NAVRepair outperformed specialized tools like VRepair (a fine-tuned model) and RING (a multilingual repair framework), achieving a 26.14% "perfect patch" rate compared to VRepair's 22.67%.

Discussion and Insights

A key finding of the study is that syntax trumps semantics: structural hints (MEN-related information) were significantly more important for a successful fix than high-level CWE descriptions. Knowing where and how the structure is broken matters more to the LLM than just knowing the name of the vulnerability.

However, NAVRepair still faces an "Out-of-Context" Wall. Failures often occur when a fix requires a variable or function name not present in the local code snippet. Because LLMs are "hallucination-prone," they struggle when they must invent external context. Future work will focus on expanding the context window to include project-wide symbols and testing the framework on memory-safe languages like Java or Python.

Ultimately, NAVRepair is model-agnostic, suggesting that the method of prompting is just as critical as the model itself for enhancing real-world C/C++ code security.

Summary updated

Abstract

The rapid advancement of deep learning has led to the development of Large Language Models (LLMs). In the field of vulnerability repair, previous research has leveraged rule-based fixing, pre-trained models, and LLM's prompt engineering. However, existing approaches have limitations in terms of the integration of code structure with error types. Besides, due to certain features of C/C++ language, vulnerability repair in C/C++ proves to be exceptionally challenging. To address these challenges, we propose NAVRepair, a novel framework that combines the node-type information extracted from Abstract Syntax Trees (ASTs) with error types, specifically targeting C/C++ vulnerabilities. Specifically, our approach employs type analysis to localize the minimum edit node (MEN) and customizes context information collection based on different error types. In the offline stage, NAVRepair parses code patches to locate MENs and designs rules to extract relevant contextual information for each MEN type. In the online repairing stage, it analyzes the suspicious code, combines it with vulnerability type templates derived from the Common Weakness Enumeration (CWE), and generates targeted repair prompts. We evaluate NAVRepair on multiple popular LLMs and demonstrate its effectiveness in improving the performance of code vulnerability repair. Notably, our framework is independent of any specific LLMs and can quickly adapt to new vulnerability types. Extensive experiments validate that NAVRepair achieves excellent results in assisting LLMs to accurately detect and fix C/C++ vulnerabilities. We achieve a 26% higher accuracy compared to an existing LLM-based C/C++ vulnerability repair method. We believe our node type-aware approach has promising application prospects for enhancing real-world C/C++ code security.
BibTeX
@misc{wang2024navrepair,
  author = {Ruoke Wang and Zongjie Li and Chaozheng Wang and Yang Xiao and Cuiyun Gao},
  title = {{NAVRepair: Node-type Aware C/C++ Code Vulnerability Repair}},
  year = {2024},
  eprint = {2405.04994},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.04994},
}
Figure