Neural models that flag vulnerable code look strong when they are tested on functions from the same software projects they were trained on. They fall apart on projects they have never seen. On the DiverseVul benchmark, F1 drops from about 49% on a random split to about 9.4% on unseen projects (DiverseVul). One diagnosis explains much of this collapse. The evidence for a vulnerability is usually local: a missing bounds check or an unchecked return value spanning a few lines. A deep transformer, which lets every token attend to every other token at every layer, is free to base its decision on project-wide cues such as naming habits, formatting, and API idioms. Those cues predict the label inside a project and mean nothing outside it.
We propose a hybrid architecture that combines a convolutional path and a transformer path. It changes where the model's decision is allowed to come from. A convolutional stream (sliding filters that only see a few neighbouring lines) produces a vulnerability score for each line. The pre-trained transformer's global context may only re-weight those line scores. It can never add evidence of its own. The claim is that this "local-evidence bottleneck" narrows the cross-project gap and keeps the model pointing at the actual buggy lines, without the data perturbations or environment clustering that current debiasing methods require.
Core Claim
The claim is testable and could fail. Take a pre-trained code encoder and force its function-level decision through a bottleneck. The bottleneck aggregates line scores computed by a small convolutional network over early-layer features and a name-abstracted view of the tokens. Global attention acts only as a gate on those scores. This model should beat the same encoder fine-tuned normally on unseen-project splits. It should also match or beat optimization-level fixes such as causal adjustment and invariant-risk penalties. And it should locate buggy lines far better when tested across projects.
The hybrid literature offers three ways to combine convolution and attention: a transformer bottleneck inside a convolutional encoder-decoder, parallel local and global streams, and convolutions interleaved inside each block. This proposal is a parallel dual-stream design with one added constraint, an asymmetric fusion in which the global stream cannot bypass the local one. Parallel dual-stream vision models such as ClassLIE fuse the two streams symmetrically. That symmetry is exactly what we must avoid, because a symmetric fusion lets the shortcut-prone global stream dominate.
What the Prior Work Establishes
Three papers motivate the proposal by documenting the gap. DiverseVul benchmarks 11 architectures and shows the unseen-project collapse. It also finds that adding training data does not fix it, which undercuts pure scaling as a remedy. Unveiling Project-Specific Bias in Neural Code Models defines Cond-Idf, a score for how strongly a token predicts the label only because of project quirks. Using attribution, it shows that CodeBERT-style models lean on project-specific identifiers. Evaluating Line-level Localization Ability shows that CodeT5 reaches F1 0.93 on Big-Vul while its Detection Alignment is only 0.077. Detection Alignment measures how much of the model's attribution falls on the truly vulnerable lines. The same paper finds that early transformer layers align with the buggy lines better than deep ones. That finding directly suggests reading local evidence from early layers, and our design does so.
Two papers supply competing remedies, and both leave the architecture untouched. CausalVul perturbs variable names and injects dead API calls to discover spurious features, then removes them at inference with a backdoor adjustment. It reports up to 200% relative improvement when trained on one project and tested on another. Batch Partition Regularization groups training samples by logic similarity and applies an invariance penalty across those groups. Our design needs neither hand-built perturbations nor environment discovery. Its restriction is structural.
Three papers relate to the method or bound it. AST-Transformer restricts attention to syntax-tree neighbours within a fixed distance and improves Java summarization BLEU from 44.58 to 46.64. It was never tested across projects, and its locality applies throughout the network rather than at the decision point. White-Basilisk is a hybrid of state-space layers, compressed attention and mixture-of-experts that reaches F1 0.291 on PrimeVul. It does not evaluate cross-project splits or shortcut reliance. A Guide to Robust Generalization finds that convolution-transformer hybrids are the most robust image classifiers in the small-parameter, low-data regime. This is suggestive, but it comes from images only, and we treat it as a hypothesis to test rather than as evidence. ConTest constrains us from another direction: feeding linearized syntax trees into a plain transformer helps on random splits (+4.1 BLEU), yet project splits still fall from 38.2 to 15.4 BLEU. Better syntactic input alone is therefore not enough, which is why we change the decision pathway rather than the input format.
Research Program
The backbone is a pre-trained code encoder already used in this literature, namely the CodeT5 encoder or UniXcoder. The same backbone is used for every compared method.
The local stream takes token features from an early layer $\ell$ of the backbone. It concatenates each feature with an embedding of an abstracted token that keeps the syntactic category, the node type in the syntax tree, and the role of the identifier (local variable, parameter, callee), but drops the identifier's spelling. The layer $\ell$ is chosen on held-out validation projects. A few depthwise-separable 1D convolutions follow, with a receptive field of roughly three to five source lines. Features are pooled within each line to give a local evidence score $s_i$ for line $i$.
The global stream is the full-depth backbone. From the whole function it produces a gate $a_i \in (0,1]$ for each line, which expresses how relevant the surrounding context makes line $i$. For example, it can learn that a buffer is attacker-controlled. The function-level logit is a soft maximum over gated local scores:
$$z = \tau \log \sum_{i=1}^{L} \exp\!\big((s_i + \log a_i)/\tau\big)$$
Here $L$ is the number of lines and $\tau$ is a temperature. Because $\log a_i \le 0$, context can only suppress or keep local evidence; it cannot create evidence. A function is flagged only if some short window of code looks locally suspicious.
This structure also gives line localization for free: the $s_i$ scores are directly comparable to Detection Alignment. We will analyze three things. The first is how much attribution each model places on high-Cond-Idf tokens. The second is how Detection Alignment changes from in-project to cross-project testing. The third is whether the gap between random-split and unseen-project F1 shrinks.
Baselines
The naive reference point is the same backbone fine-tuned with a standard pooled classification head, using the same data, epochs and tuning budget. It is the model that collapses to about 9.4% F1 on DiverseVul unseen projects in DiverseVul, and we will re-measure it under our exact split. For context, the best full-dataset model in that paper, NatGen, reports F1 of 47.15% on DiverseVul, and every other gain is measured against these numbers.
The strongest existing method aimed at this exact question is CausalVul. We will run it on the same backbone with its variable and API perturbations, training on the same projects. Its reported gains are relative improvements ("up to 200%" out of distribution); no reported absolute cross-project F1 was found on Lacuna for CausalVul on DiverseVul. We will also run the invariance penalty from Batch Partition Regularization, whose reported figure is on a different task: 80.57% versus 66.01% type-inference accuracy under renaming. As the strongest architectural competitor we will run White-Basilisk, with its reported PrimeVul F1 of 0.291 as a reference. That figure is not cross-project, so we will retrain it on our project splits.
Every method gets the same number of hyperparameter trials, selected on validation projects disjoint from both the training and test projects. Selecting on validation projects matters because selecting on in-project validation would quietly reward shortcuts.
Beating the field means two things on the unseen-project split, averaged over at least three project partitions with seeds and confidence intervals. The proposal must improve F1 over the fine-tuned backbone by a margin outside seed variance, and it must match or beat CausalVul. It must do this while raising cross-project Detection Alignment, not trading it away.
Ablations isolate the ingredients. The first keeps the convolutional stream but adds the global vector to the logit, which removes the bottleneck. The second replaces the convolution with per-line pooling of deep-layer features inside the same soft-maximum bottleneck. The third drops the abstracted-token channel. The fourth reads the local stream from the last layer instead of an early one.
Why not just…?
Why not just scale the model or the data? DiverseVul reports that more training data did not further improve detection, and its unseen-project collapse holds across all 11 architectures. White-Basilisk outperforms a far larger StarCoder2 model, which suggests architecture matters more than size. We will still include a larger backbone fine-tuned normally as a check.
Why not just canonicalize identifiers or augment with renamings, then fine-tune normally? This is a real threat, because much of the shortcut signal lives in names. The evidence against augmentation alone is indirect. Renaming-based adversarial training reached only 66.01% where the invariance penalty reached 80.57% in Unveiling Project-Specific Bias. And CausalVul finds that models keep finding other non-causal cues, such as dead API calls. Canonicalization also destroys informative names like length variables. We test this directly by fine-tuning the backbone on fully canonicalized code. If that model matches our full model on unseen-project F1 and Detection Alignment, then the architectural claim fails and we will report that the abstraction, not the hybrid, did the work.
Why not just read predictions out of early layers, given that they align better? The finding in Evaluating Line-level Localization Ability makes this the most natural simple alternative. We run an early-layer linear readout, and also a per-line soft-maximum over early-layer features with no convolution and no gate. If either one matches the full model, the contribution shrinks to "bottleneck over early features," and we will say so. Our expectation is that early features alone lack the context needed to reject locally suspicious but safe code, which is the job the gate does.
Why not just use a graph network or distance-limited tree attention? DiverseVul finds that ReVeal-style graph networks underperform pre-trained transformers. AST-Transformer has never been evaluated across projects. We include tree-restricted attention on the same backbone as a baseline, to separate "local operators everywhere" from "local operators at the decision."
Proposed Experiments and Decisive Checks
The primary benchmark is DiverseVul with unseen-project splits, following DiverseVul. Because that paper estimates fix-commit labels at only about 60% accuracy, we confirm every result on PrimeVul, which was curated to remove duplication and labeling noise. We add leave-project-out splits of Big-Vul, which has line-level fix information for measuring Detection Alignment with the vuln-localization-eval tooling. Devign serves as a secondary check.
The metrics are function-level F1, the ratio of unseen-project F1 to random-split F1, Detection Alignment computed on both in-project and cross-project test sets, and the share of attribution mass that falls on the top-decile Cond-Idf tokens.
The decisive check has three parts that must agree. On unseen projects, the full model beats the fine-tuned backbone and at least ties CausalVul. Its cross-project Detection Alignment clearly exceeds the 0.07–0.13 range reported for baselines in Evaluating Line-level Localization Ability. And the no-bottleneck ablation loses most of both gains.
We will also test whether the method is complementary to existing fixes by stacking CausalVul's adjustment onto our model. If it adds nothing, the two methods probably remove the same shortcuts. If F1 improves while alignment does not, the model has found a new shortcut through the gate, and we will probe that directly by measuring whether the gates $a_i$ correlate with high-Cond-Idf tokens.
References
- DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection
- ClassLIE: Structure- and Illumination-Adaptive Classification for Low-Light Image Enhancement
- Unveiling Project-Specific Bias in Neural Code Models
- Evaluating line-level localization ability of learning-based code vulnerability detection models
- Towards Causal Deep Learning for Vulnerability Detection
- AST-Transformer: Encoding Abstract Syntax Trees Efficiently for Code Summarization
- White-Basilisk: A Hybrid Model for Code Vulnerability Detection
- A Guide to Robust Generalization: The Impact of Architecture, Pre-training, and Optimization Strategy
- ConTest: A Unit Test Completion Benchmark featuring Context