Many real security bugs span several functions. A buffer is allocated in one function, indexed in another, and freed in a third. To catch such a bug, a neural detector has to read the target function together with its callers and callees. Two things prevent that. First, a Transformer's self-attention compares every token with every other token, so its cost grows with the square of input length, and whole call chains quickly get too long. Second, labeled vulnerable code is scarce and noisy, so the model cannot learn from a few thousand examples which of those thousands of tokens matter. Prior work found that feeding more raw context to a model often adds noise rather than signal.
We propose to solve both problems with one mechanism. A small, fixed budget of full-detail tokens is allocated across the whole call chain. The choice of which tokens get that budget is not learned from scarce vulnerability labels. It is driven by two sources of supervision that are free and abundant: the textual definitions of weakness categories (the standard Common Weakness Enumeration, or CWE, entries, such as "out-of-bounds write") and cheap data-flow paths to memory and resource operations. All other tokens are merged into coarse summary tokens. The central claim is that, at equal compute, this rule for choosing tokens preserves the statements that trigger a vulnerability much better than the model's own attention scores do. It should also raise detection on cross-function bugs and on unseen projects, which are the two settings where current detectors fail most badly.
Core Claim and What Is New
The evidence that context matters is already strong. On the Effectiveness of Function-Level Vulnerability Detectors for Inter-Procedural Vulnerabilities builds InterPVD, 769 annotated vulnerabilities, and finds that 24.3% span functions, on average 2.8 call layers deep. It also reports that five deep detectors lose an average of 18.7% F1 when evaluated on the function where the bug triggers rather than the function that was patched. VulEval shows that relevant dependency context improves ChatGPT's detection by up to 26%. Context-Enhanced Vulnerability Detection Based on Large Language Model (PacVD) shows the other side of the problem. It compresses callee context with hand-designed abstraction levels around primitive APIs such as malloc and free, and adding the finest level, concrete variables obtained through pointer-alias analysis, made results worse. These three papers motivate the proposal: context helps only when irrelevant tokens are strongly suppressed.
The method borrows from vision. SG-Former keeps salient image regions at fine resolution and merges background regions into coarse tokens using a significance map. FastV drops low-attention visual tokens after layer 2 and cuts FLOPs by up to 45%. Neither has been applied to code. Both rank tokens by signals the network learns end to end, and that is exactly what fails when labels are scarce and the decisive difference is a single token. The supervision for choosing tokens comes from CLeVeR, which aligns code with natural-language vulnerability descriptions using a contrastive loss and lets the description attend over code tokens to filter out ordinary functional logic. CLeVeR does this only within one function, using dense attention.
The new step is to use the description-to-code relevance signal, together with a parse-only data-flow mask, as the allocator of a token budget that spans functions. This turns CLeVeR's filter into the mechanism that makes cross-function context affordable. PacVD compresses context outside the model with fixed rules. Here, compression happens inside the model, at the level of individual tokens, and depends on the weakness category being tested. Unlike FastV or SG-Former, the ranking is anchored to security semantics rather than to the network's own attention.
Research Program
The input is a target function plus its callers and callees up to depth three. Call edges come from VulEval's dependency annotations or from a lightweight parser, and nothing needs to compile. This matters because disclosed vulnerable code often fails to build (Kim Le et al.).
The encoder is a pre-trained code model of the CodeT5 or UniXcoder family. Its lower layers run separately inside each function, so their cost is quadratic only in each function's own length. At the cut layer, every token receives a relevance score that combines two parts. The first is cross-attention from embeddings of the CWE definition texts, used as queries. At test time, when no description exists, it uses all CWE definitions or CLeVeR's description simulator. The second is a binary flag for whether the token lies on a syntactic definition-use path that reaches a primitive memory, index, or resource operation. This is a cheap, parse-level version of the data-flow abstraction in DeepDFA.
The top $B$ tokens across the whole chain are kept at full detail. Every remaining statement block is averaged into one coarse token, following SG-Former's importance-guided aggregation. The upper layers then apply full softmax attention over this compressed sequence. For functions of lengths $n_1,\dots,n_F$ and $M$ coarse tokens, the cost of the attention scores falls from $(\sum_i n_i)^2$ to
$$\sum_i n_i^2 + (B+M)^2,$$
because only within-function attention and attention over the compressed sequence remain. We keep softmax attention rather than linear attention for the global stage. Why Softmax Attention Outperforms Linear Attention finds that linear variants struggle to isolate sharp, localized dependencies, which is exactly the "one unchecked index" pattern this task depends on.
Training combines the binary detection loss with an InfoNCE alignment loss on code–description pairs from VCLData, CLeVeR's set of 280,034 described functions. The description encoder therefore learns its relevance signal from text rather than from the scarce cross-function labels. The analysis component measures how much of each InterPVD triggering statement survives selection under each rule, and how this survival rate relates to detection accuracy.
Baselines
The naive reference is the same encoder applied to the target function alone, with no context. This reproduces the function-level myopia that InterPVD documents. The second reference is the same encoder given the full call chain, truncated to the maximum length the compute budget allows. This shows what raw context buys before any selection.
The strongest existing method on the cross-function question is PacVD. On NVD-New it reports 69.09% F1 with DeepSeek-R1, chain-of-thought prompting, and abstraction level A3, which is 9.25 points above its closest baseline (Context-Enhanced Vulnerability Detection Based on Large Language Model). That result uses a far larger model, so we compare against it in two ways. We report the published number as a ceiling to approach. We also run a fair version: PacVD's A3 abstracted context fed to our same-size encoder, fine-tuned on the same data with the same tuning budget. Beating the fair version at equal FLOPs, and coming within a few points of the 69.09% obtained with a much larger model, would count as success.
On single functions, CLeVeR reports 72.82% F1 on RealData (CLeVeR). Our model must not fall measurably below this when the context is empty, which checks that compression does no harm. For generalization to new codebases, the reference is the unseen-project result in DiverseVul: roughly 49% F1 on random splits collapses to about 9.4%. For complex real bugs, DeepDFA detects 8.7 of 17 DbgBench bugs, while LineVul and UniXcoder detect none. On VulEval's repository-level time split, no reported F1 for a comparable trained detector was found on Lacuna, so we will establish that baseline ourselves.
The key ablation holds the budget $B$, the encoder, and the training data fixed and changes only the selection rule. The rules are random selection, FastV-style ranking by attention received at the cut layer, EViT-style ranking by attention from the class token, lexical-overlap ranking against the target function, the data-flow mask alone, the CWE-description relevance alone, and the combination of both.
Why not just…?
Why not just slice the code with static analysis and feed the slice to a standard model? This is PacVD's route, and it is a serious competitor. However, its own ablation shows that finer static resolution through alias analysis hurt performance. The static tools reported in InterPVD locate triggering statements poorly: Checkmarx reaches 12.7% and Flawfinder 9.8%, against 69.8% for the purpose-built VulTrigger. Fixed rules also cannot adapt to the weakness category being tested. The fair PacVD-on-our-encoder baseline settles this question directly. If it matches our method at equal compute, the in-model allocator adds nothing, and we will report that.
Why not just use a graph neural network, whose cost grows only with the number of edges? DeepDFA is fast and generalizes surprisingly well on DbgBench. Graph models, however, lost to pre-trained language models as data scaled in DiverseVul, and Devign had to discard functions larger than 500 nodes. Our method keeps DeepDFA's useful structural prior as the data-flow mask, and the data-flow-only ablation measures how much of the gain that prior supplies on its own. We also run DeepDFA with call edges added to its graphs, as a graph baseline that sees cross-function structure.
Why not just apply Top-K or FastV pruning off the shelf? This is the most dangerous alternative. Which Tokens to Use? found that simple Top-K pruning and EViT match or beat elaborate dynamic schemes across 10 methods. The reason to expect code to differ is that FastV relies on attention concentrating on instruction anchors, whereas vulnerabilities often hinge on a single, rare token (Li et al.). The selection-rule ablation and the trigger-survival measurement test this directly. If attention-based Top-K keeps triggering statements as often as description-guided selection does, the specialized allocator is unnecessary and the paper becomes a negative result about selection rules.
Why not just retrieve the most similar callees lexically and truncate? VulEval found that lexical retrieval beat semantic retrieval for gathering dependencies. Lexical retrieval, however, selects whole functions, not tokens inside them. The lexical-ranking ablation at equal token budget shows whether finer, token-level allocation matters.
Proposed Experiments
Training uses VulEval's training portion and DiverseVul, with VCLData supplying the contrastive pairs. Evaluation covers four settings: InterPVD for cross-function and trigger-function detection, VulEval's time split for repository-level detection, DiverseVul's unseen-project split, and NVD-New against PacVD. We report F1, attention FLOPs, and wall-clock time. Wall-clock time matters because merged sequences of variable length can erase theoretical savings, a problem FastV leaves open, so we pad merged sequences to fixed budgets.
The first decisive check is diagnostic and inexpensive. Using InterPVD's annotated triggering statements, we measure the fraction of triggering tokens retained under each selection rule at budgets from 5% to 30% of the chain. If description- and data-flow-guided selection does not retain clearly more triggering tokens than attention-based Top-K, the central claim fails before any large training run.
The second check is the trigger-versus-patch gap. With context and guided selection, the 18.7% F1 drop that InterPVD reports should shrink substantially. The third check is the unseen-project split, which tests whether context focused on security semantics reduces memorization of project-specific coding idioms. Any gain over the roughly 9.4% F1 baseline at equal compute would be meaningful. We will also check how much DiverseVul's roughly 60% label accuracy caps the achievable score, by reporting results on the manually verified Devign subset alongside.
References
- On the Effectiveness of Function-Level Vulnerability Detectors for Inter-Procedural Vulnerabilities
- VulEval: Towards Repository-Level Evaluation of Software Vulnerability Detection
- Context-Enhanced Vulnerability Detection Based on Large Language Model
- SG-Former: Self-guided Transformer with Evolving Token Reallocation
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models
- CLeVeR: Multi-modal Contrastive Learning for Vulnerability Code Representation
- Maximal Divergence Sequential Autoencoder for Binary Software Vulnerability Detection
- DeepDFA: Dataflow Analysis-Guided Efficient Graph Learning for Vulnerability Detection
- Why Softmax Attention Outperforms Linear Attention
- DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection
- Which Tokens to Use? Investigating Token Reduction in Vision Transformers