TL;DR
VulAgentRL is an agentic reinforcement learning (RL) framework that detects complex, multi-function software vulnerabilities by navigating a Code Property Graph (CPG). By utilizing the CPG as both an interactive tool at inference and an exact evidence verifier during training, it prevents the common RL pitfall of "guessing" and significantly outperforms frontier LLMs.
What Problem Does This Paper Address?
Most learning-based vulnerability detectors evaluate functions in isolation. However, real-world security flaws rarely stay confined to a single block of code. The authors find that 71.7% of vulnerable functions require interprocedural evidence—context from callers, callees, or dataflow paths outside the function body—to be correctly diagnosed.
While large language models (LLMs) acting as autonomous agents could theoretically navigate codebases to find this evidence, training them via reinforcement learning is notoriously difficult. If you only reward the agent based on whether its final verdict ("vulnerable" or "safe") is correct, the model suffers from evidence-free degeneration. Instead of actually investigating, the agent learns to exploit class priors and simply guess the answer.
Methods
To solve this, the authors introduce VulAgentRL, which utilizes a Code Property Graph (CPG) in a dual role: as the agent's navigation tool and as the training environment's automated verifier.
The training pipeline consists of two main stages:
- Reasoning-and-Tool-Use SFT: The model (initialized from Qwen2.5-Coder-7B-Instruct) is first trained via Supervised Fine-Tuning on high-quality, multi-turn investigation trajectories distilled from a frontier teacher model (Claude Opus).
- GRPO RL with a CPG Verifier: The SFT model is optimized using Group Relative Policy Optimization (GRPO). Crucially, because every node in the CPG has a persistent integer identifier, the reward function can exactly match the agent's cited evidence nodes against the ground truth.
The composite reward function is strictly rule-based and balances three objectives: Verdict Agreement (60%), Evidence Grounding via $F_1$ score of node IDs (20%), and Common Weakness Enumeration (CWE) class agreement (20%).
Datasets
- PrimeVul: The primary corpus used for training and evaluation, containing 1,122 unique functions across 275 repositories. The authors apply a strict repository-level 8:1:1 split and per-commit deduplication to completely eliminate data leakage.
- TitanVulOOD: A newly curated, clean out-of-distribution (OOD) test set containing 648 C/C++ functions across 217 repositories entirely disjoint from PrimeVul to evaluate cross-project generalization.
Key Insights and Ideas
- The CPG as an Exact Verifier: Evaluating an agent's reasoning using an "LLM-as-a-judge" or fuzzy text matching is brittle and unreliable. By leveraging the persistent integer IDs of a CPG, evidence verification becomes a deterministic set-comparison problem.
- SFT Warm-Starts are Structurally Mandatory: The authors demonstrate that RL cannot bootstrap tool-use behavior from a cold start. If a base model starts with zero tool calls, the group-relative advantage in GRPO provides no gradient signal to steer the policy toward querying. SFT initialization is required to bridge this gap.
- RL Breeds Efficiency: Interestingly, reinforcement learning didn't just make the model more accurate; it made it more efficient. The RL-tuned model dropped its average number of tool calls per rollout from 7.6 to 4.3, learning to conduct shorter, better-targeted investigations.
Key Results
VulAgentRL achieves state-of-the-art performance, outperforming both massive frontier models and specialized agent baselines on the strict Pair-Wise-Correct (P-C) metric (which requires classifying both the vulnerable and patched versions of a function correctly).
- Performance Leap: On the PrimeVul test set, VulAgentRL achieves a P-C score of 0.378, outperforming Claude Opus 4.7 (0.273) and the previous state-of-the-art agentic detector, JitVul (0.051).
- OOD Generalization: On the unseen TitanVulOOD dataset, VulAgentRL maintains its edge with a P-C score of 0.302, proving the graph-grounded training transfers well to entirely new codebases.
- Class Imbalance Robustness: In realistic scenarios where benign code heavily outnumbers vulnerable code (up to a 1:16 ratio), VulAgentRL consistently maintains a higher F1-score than its SFT and 0-shot counterparts.
Limitations and Future Work
While the underlying CPG substrate is inherently language-agnostic, the authors limited their current empirical evaluation strictly to C/C++ codebases. Additionally, VulAgentRL is currently optimized purely for vulnerability detection. The authors note that extending this graph-verified agentic framework to automated vulnerability repair (patching) is a compelling next step.
Summary updated