Learning to Focus: Context Extraction for Efficient Code Vulnerability Detection with Language Models

Lorenzo CavallaroarXiv 2025 · DOI · arXiv

TL;DR

This paper introduces FocusVul, a novel framework that helps large language models (LLMs) more effectively detect code vulnerabilities. By intelligently selecting and condensing relevant code snippets, FocusVul significantly boosts detection accuracy and efficiency, overcoming the limitations of LLMs struggling with long, complex code.

What Problem Does This Paper Address?

Modern software development relies heavily on Large Language Models (LLMs) for various tasks, including code vulnerability detection. However, LLMs face significant challenges when analyzing real-world codebases. These challenges stem from two main issues:

  1. Sparse and Uncertain Vulnerability Locations: Vulnerabilities often exist in very specific, small parts of a function. LLMs have strict input length limits, meaning that when analyzing long functions, they might truncate the code, potentially missing the vulnerable section or including irrelevant code, leading to misleading training signals.
  2. Sparsity of Vulnerabilities: Vulnerabilities are rare events in code. This low signal-to-noise ratio makes it difficult for LLMs to learn effectively, as they might pick up on spurious patterns instead of genuine vulnerability indicators.

To address this, researchers have explored extracting "context" – relevant code snippets – to feed into LLMs. However, existing methods often rely on predefined rules (like specific CWE types) or are unavailable during inference (like commit history). This paper aims to bridge this gap by developing a model-agnostic framework that learns to identify and extract concise, vulnerability-relevant context for efficient and accurate LLM-based vulnerability detection.

Methods

FocusVul employs a two-stage, model-agnostic framework:

  1. Learning-based Vulnerability-Relevant Region (VRR) Identifier:
    • Commit-based Supervision: The core idea is to leverage vulnerability-fixing commits. These commits explicitly highlight lines of code that were changed to fix a vulnerability (VRRcom).
  2. Hierarchical Semantic Modeling: To generalize from these commit-based examples to unseen code, FocusVul uses a hierarchical approach. It models semantics at the token, line, segment (overlapping chunks of lines), and function levels. This allows it to capture context beyond individual lines.
  3. Semantic Generalization Masking: To prevent models from overfitting to specific keywords, FocusVul applies asymmetric masking. Lines identified as part of a VRR are masked at a higher rate than other lines, encouraging the model to learn from broader contextual and functional cues.
  4. Fallback to Heuristics: Since commit data isn't available during inference, the trained VRR identifier predicts VRRs. If prediction is uncertain, it falls back to heuristic-based VRRs (e.g., sensitive API calls).
  5. Sensitive Context Extractor:
    • Structure-Aware Slicing: Once VRRs are identified, this module extracts context around them. It uses both dependency relationships (data and control dependencies) and execution relationships (basic block sequences) to capture semantic and structural information.
  6. Minimal Enclosure: The extraction process aims to be minimal, including only essential adjacent basic blocks to preserve local execution semantics while keeping the context concise.
  7. LM-Friendly Input: The extracted context is reordered to maintain syntactic coherence, creating compact inputs suitable for LLMs.

Datasets

  • Vulnerability Detection Dataset: Experiments were conducted on PrimeVul, a large real-world C/C++ vulnerability dataset. It unifies data from benchmarks like BigVul, CrossVul, CVEfixes, and DiverseVul. It includes both individual and paired (vulnerable/fixed) samples.
  • VRR Identification Dataset: A subset of 6,443 pairwise samples from the PrimeVul training split was used to train the VRR identification module, leveraging commit-based line changes as supervision.

Key Insights and Ideas

  • Commit Data as a Powerful Signal: Vulnerability-fixing commits provide highly precise, fine-grained supervision that can be learned and generalized.
  • Hierarchy is Key for Context: Capturing semantic context at multiple levels (token, line, segment, function) is crucial for understanding complex code and identifying subtle vulnerabilities.
  • Asymmetric Masking for Robustness: Applying different masking rates to VRR lines and non-VRR lines helps models generalize better and rely less on memorizing specific keywords.
  • Dependency and Execution Flows Enhance Context: Combining data/control dependencies with basic block sequences provides a richer, more informative context than relying on just one type of relationship.
  • Conciseness Boosts Efficiency: By extracting compact, relevant code snippets, FocusVul significantly reduces the computational load on LLMs without sacrificing accuracy.

Key Results

FocusVul demonstrated significant improvements across various LLMs:

  • Detection Effectiveness: Achieved an average 164.04% improvement in F1-score compared to function-level fine-tuning.
  • Efficiency: Reduced average fine-tuning FLOPs by 19.12%.
  • Compression: Achieved higher Compression Rate (CR) and Retention Rate (RR) compared to other context selection strategies, meaning more input reduction with better preservation of important tokens.
  • Pair-wise Differentiation: Significantly improved the ability of models to distinguish between vulnerable and benign code pairs, indicating better understanding of vulnerability signals.
  • Ablation Studies: Confirmed the importance of commit-based supervision (VRRcom) and the learned VRR prediction mechanism. Removing these components led to substantial performance drops.

Limitations and Future Work

  • Reliance on Commit Data: FocusVul's training relies on commit annotations, which are not always available. While the framework learns to generalize, the availability and quality of commit data are still important. The authors note that this reflects the nature of commit-supervised learning and suggest broader data collection as a future direction.
  • Real-world Challenges: Despite improvements, real-world vulnerability detection remains challenging for LLMs. Further research is needed to enhance robustness and precision.
  • Future Work: The authors suggest exploring more systematic data collection for commit annotations and further research into improving LLM robustness for vulnerability detection.

Summary updated

Abstract

Language models (LMs) show promise for vulnerability detection but struggle with long, real-world code due to sparse and uncertain vulnerability locations. These issues, exacerbated by token limits, often cause models to miss vulnerability-related signals, thereby impairing effective learning. A key intuition is to enhance LMs with concise, information-rich context. Commit-based annotations offer precise, CWE-agnostic supervision, but are unavailable during inference, as they depend on historical code changes. Moreover, their extreme sparsity, often covering only a few lines, makes it difficult for LMs to process directly. In this paper, we propose FocusVul, a model-agnostic framework that improves LM-based vulnerability detection by learning to select sensitive context. FocusVul learns commit-based annotation patterns through hierarchical semantic modeling and generalizes them to identify line-level vulnerability-relevant regions during inference. It then extracts LM-oriented context via both dependency and execution flows surrounding selected regions, yielding semantically rich inputs for effective vulnerability detection. Experiments on real-world benchmarks show that FocusVul consistently outperforms heuristic-based and full-function fine-tuning approaches, improving classification performance by 164.04% and reducing FLOPs by 19.12% on average.
BibTeX
@misc{cavallaro2025learning,
  author = {Lorenzo Cavallaro},
  title = {{Learning to Focus: Context Extraction for Efficient Code Vulnerability Detection with Language Models}},
  year = {2025},
  eprint = {2505.17460},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.17460},
}
Figure