Reading Code With Weakness Definitions

Most learned vulnerability detectors repeat a key weakness of signature scanners. They are trained to answer one fixed question, "is this function buggy, yes or no?", sometimes with a closed menu of weakness types. They therefore cannot say anything useful about a kind of flaw they were not trained on. They also flatten two kinds of graded information into hard bins. The first is how severe a flaw is. The second is how related two weakness types are: an out-of-bounds write is a close cousin of a buffer over-read and has nothing to do with SQL injection. This proposal makes the detector read code through the written definitions of the MITRE Common Weakness Enumeration (CWE), the standard catalogue of software weakness types. Each definition is encoded as text and used as a query that searches the function's tokens for matching evidence. A weakness type the model never saw labelled examples of can then still be checked, provided its definition can be written down.

The concrete bet is this. A code model whose output layer is built from per-weakness cross-attention over definition text will detect held-out weakness types in unseen projects better than an identical model whose weakness slots are anonymous learned vectors. Training it jointly on three things should make it more useful for triage than a binary detector: graded weakness targets, line localization read directly from its attention maps, and a continuous severity score. The test that decides the first claim is cheap. If the definition text is shuffled across slots and the gain disappears, the semantics are doing the work. If not, they are not.

What exists and where this differs

Three lines of work motivate the idea. The first shows that the problem is real and severe. DiverseVul assembles 18,945 vulnerable functions over 150 CWEs and 797 projects. Its best model, NatGen, reaches 47.15% F1 on the full dataset. Models near 49% F1 on random splits fall to about 9.4% on unseen projects. That collapse is the learned-model version of the closed-world failure that signature scanners show against zero-days.

The second line supplies the multi-task recipe and sets the bar we must clear. VulLLM instruction-tunes CodeLlama-13B jointly on three tasks: detection, line localization, and natural-language explanation. It reports 66.54% overall F1 across six C/C++ datasets and an F1 8.58% higher than UniXcoder on four out-of-distribution sets. It also reports a 9.3% F1 drop when the auxiliary tasks are removed. VulLLM still decodes free text against a binary target and never conditions on what a weakness is. We keep its finding that auxiliary localization helps. We replace its separate localization output with the attention maps of the weakness queries themselves, so the same mechanism that scores a weakness also points to the lines that justify it. Devign shows that structural program graphs help, but it is a single-task binary classifier and constrains us only as a possible alternative encoder.

The third line supplies the mechanism. The label-aware BERT attention network for zero-shot multi-intent detection uses encoded label descriptions as cross-attention queries over token states, inside a jointly trained intent-and-slot model, to recognize unseen intents in speech-assistant utterances. CrossTune shows that cross-attention against expanded label definitions beats prompt search for few-shot text classification. The CVE-to-CWE mapping study shows how much CWE semantics carry in security text. Fine-tuned Sentence-BERT reaches a mean reciprocal rank (MRR) of 0.9142 when ranking CWEs for CVE advisory text, against 0.1514 for keyword retrieval. That work reads human-written disclosures after the fact and never inspects code. Taken together, these papers show that definition-queried attention works on natural language and that CWE text is semantically rich. None of them asks whether a definition can find the evidence for a weakness inside a function body. That transfer is the new part, and it is not obvious it will work, because code tokens and English definitions share little vocabulary.

The discretization literature sets the evaluation lens rather than the method. Work on regression-based biomarker prediction and on creatinine clearance in cardiac surgery risk shows that binary cutoffs blind models to cases just below the threshold. We test the security analogue directly. A detector trained on "vulnerable or not" should rank severity poorly among the functions it flags, and worst near triage boundaries.

Core Claim

Let $h_1,\dots,h_N$ be the token states of a function from a pretrained code encoder. Let $q_c$ be the encoding of the definition text of weakness $c$ under a frozen-then-tuned text encoder. The model computes, for every weakness in the catalogue,

$$a_c = \mathrm{softmax}_i\!\left(\frac{(W_q q_c)^\top (W_k h_i)}{\sqrt{d}}\right), \qquad s_c = w^\top \sum_i a_{c,i}\, W_v h_i ,$$

Here $a_c$ is a distribution over the $N$ tokens and $s_c$ is a scalar evidence score for weakness $c$. The weight matrices $W_q, W_k, W_v$ and the vector $w$ are shared across all weaknesses. Because nothing is indexed by $c$ except the definition, a new weakness costs one text encoding and no new parameters. The function-level vulnerability score is the maximum of $s_c$ over the catalogue. A function is flagged as matching no known weakness when every $s_c$ is low but a weakness-agnostic "anomalous" score stays high. That second score is a learned generic query with no definition attached. This gives the detector an explicit way to say "suspicious, but not like anything I know."

Research Program

Three supervision signals are trained together. The first is graded weakness targets. A function labelled with CWE $c$ receives full credit on $c$ and partial credit on its parent and sibling CWEs, scaled by distance in the CWE hierarchy. Other functions receive zero. This replaces one-hot targets that pretend all wrong weaknesses are equally wrong.

The second is localization through attention. For functions whose fix commit is available, the attention map $a_c$ of the labelled weakness is pushed toward the lines changed by the fix, using a cross-entropy term between $a_c$ and a normalized indicator over changed lines. DiverseVul's audit finds the fix-commit heuristic only about 60% accurate, because unrelated refactoring is bundled into security commits. We therefore down-weight this term, and we evaluate localization only on Juliet, whose good and bad variants are cleanly paired by construction.

The third is continuous severity. For functions linked to a CVE record, a small regression head on the pooled weakness representation predicts the National Vulnerability Database base score on its 0–10 scale. We still need to verify how many DiverseVul and CVEfixes functions can be joined to scored records. Any shortfall goes into the report rather than being imputed.

Loss weights are chosen by validation grid under a fixed budget that is identical for every variant. We track per-task validation curves to detect the negative transfer the multi-task literature warns about. The program then asks four questions in order. Does the model work on seen weaknesses at least as well as a plain head? Does it generalize to held-out weaknesses? Is that generalization caused by the definition text? Does continuous severity training improve ranking without hurting detection?

Baselines

The naive reference point is the same pretrained code encoder (UniXcoder-scale, so that every variant is affordable to tune) with a pooled classification token and a binary head, trained on the same DiverseVul training split. Its held-out-weakness performance is whatever its binary score achieves on functions of weakness types it never saw. A definition-based model that cannot beat this point has no reason to exist.

The strongest existing method is VulLLM. On its shared out-of-distribution datasets (BigVul, CVEfixes, ReVeal, Juliet) we will quote its reported 66.54% overall F1 and its reported 8.58% out-of-distribution margin over UniXcoder. We will not re-run the 13B model under our budget, so those figures are context for comparing across scales. For the fair comparison we reproduce its three-task recipe, including its random identifier substitution, on a backbone of matched parameter count, with the same data and the same tuning budget. On held-out weaknesses we found no reported result on Lacuna for VulLLM or any code model, so that comparison is new. Beating VulLLM means two things. On seen weaknesses we must match it within noise. On held-out weaknesses in unseen projects we must exceed it in area under the ROC curve, with confidence intervals across three seeds and three different held-out weakness partitions that do not overlap.

The key ablation is a twin model with identical architecture and training in which each $q_c$ is a free learned vector rather than encoded definition text. Held-out weaknesses get freshly initialized vectors, which are effectively random. A second ablation keeps definitions but shuffles them across weakness slots. A third uses only the short CWE name instead of the full definition. A fourth removes the graded targets in favour of one-hot. A fifth removes the severity head. A sixth trains localization as a separate line-labelling output rather than through the attention maps.

Why not just…?

Why not just fine-tune a code model with an ordinary multi-task head, as VulLLM does? On seen weaknesses this may well tie our model, and we expect it to. A fixed head, however, has no slot for a weakness absent from training. Its only route to a held-out weakness is the generic binary score, which is exactly the closed-set behaviour that collapses to about 9.4% F1 on unseen projects in DiverseVul. The held-out-weakness split settles this. If the reproduced VulLLM-style model's binary score detects held-out weaknesses as well as our per-weakness queries do, then the definitions add nothing for detection, and our contribution shrinks to interpretable, labelled flags.

Why not just embed code and definitions separately and compare them by similarity, as the CVE-to-CWE work does with Sentence-BERT? This is the most serious threat. A two-tower model, which scores code against each definition with a single dot product, is cheaper and carries the same semantic content. Its 0.9142 MRR, however, was earned on English advisories, where text matches text. In code, the evidence for a weakness is often a few lines, such as a missing bounds check, inside an otherwise ordinary function, and a single pooled code vector may wash that signal out. We run a two-tower variant with the same encoders and budget. If it matches cross-attention on held-out weaknesses, we adopt it and report that cross-attention is unnecessary. What would remain is localization, which a two-tower model cannot provide without a separate output.

Why not just prompt a large language model with the CWE definitions and the code? We include zero-shot and few-shot prompting of an open instruction-tuned code model with the same definition text in the prompt. The evidence in the file is unfavourable. VulLLM attributes much of the weakness of code models to reliance on identifiers and style, and CVE-LLM needed ontology enrichment and supervised tuning to curb hallucinated entities even on advisory text. Prompting also gives no calibrated per-weakness score for ranking at a fixed false-positive rate. This baseline is reported rather than argued away.

Why not just smooth the one-hot targets along the CWE hierarchy and skip the definitions? Hierarchy-aware smoothing captures part of the "graded relatedness" benefit at no cost, so it appears as an ablation. It cannot reach a weakness with no training examples, because smoothing only redistributes credit among slots that already have trained parameters. If smoothing alone recovers most of the seen-weakness gain, we will say the graded targets, not the queries, explain that part.

Why not just fix the noisy labels first? Label noise may be the dominant cause of the cross-project collapse, and no architecture fixes it. We do not claim otherwise. We guard against noise driving the conclusions by reporting results on Juliet's clean pairs and on Devign's manually annotated functions as well as on DiverseVul. Any effect that appears only on the noisy set is treated as unconfirmed.

Proposed Experiments

Training uses DiverseVul with two kinds of holdout. Projects are split so that test projects are unseen. Weaknesses are split by holding out entire CWE subtrees, for example the injection family or the integer-error family, so that no function of those types is seen in training. Three disjoint partitions are used. Evaluation covers seen-weakness and held-out-weakness detection on unseen DiverseVul projects, transfer to BigVul, CVEfixes, ReVeal and Juliet as in VulLLM's suite, and binary detection on Devign's four projects.

Detection is measured by F1 and by area under the ROC curve. We also report recall at a fixed low false-positive rate, the operating regime that EMBER uses for static malware detection, where LightGBM reports a 93% detection rate at 0.1% false-positive rate. That figure comes from a different domain and is not a target here. Weakness identification is measured by MRR over the full catalogue, including held-out types. Localization is measured by top-k line hit rate on Juliet. Severity is measured by Spearman correlation with base scores among true positives, and by pairwise ranking accuracy restricted to pairs straddling the 7.0 high-severity boundary. Open-set behaviour is measured by how well the "anomalous but unmatched" signal separates held-out-weakness functions from clean ones when the held-out definitions are deliberately removed from the catalogue.

Three outcomes would decide the project. The first is that the definition-queried model beats its learned-query twin on held-out weaknesses by a margin larger than seed variance, while the shuffled-definition control falls back to the twin's level. Without that pattern the central claim fails, whatever the headline numbers. The second is that severity training improves boundary-pair ranking over ranking by the binary model's confidence without lowering detection F1. If it lowers F1, that is a measured case of the continuous-versus-discrete negative transfer, and we report it as such. The third is that the cross-attention model either beats the two-tower variant on held-out weaknesses or we adopt the simpler model. The project is designed so that every one of these outcomes, including the negative ones, is a publishable answer to whether written weakness knowledge can stand in for labelled examples in code.