Sparse Cross-Attention for the Second Imbalance in Defect Prediction

Defect predictors face class imbalance twice. The first imbalance is the familiar one: buggy commits or functions are rare, sometimes one in two thousand in real repositories (On Distribution Shift in Learning-based Bug Detectors). The second gets less attention. Inside each buggy commit, the fault usually sits in a few statements and one or two data or control dependencies, surrounded by dozens of unchanged, harmless ones. The standard fixes, such as focal loss and per-sample weights, act only on the first imbalance, at the output layer. When a model averages every token or graph node into one vector, the few buggy statements get diluted before the classifier ever sees them. We think this explains why output-layer reweighting stalls: on a 1:34 Java vulnerability set, focal loss plus sample weighting moved a fine-tuned code language model only from 0.86 to 0.878 ROC AUC (Finetuning Large Language Models for Vulnerability Detection).

We propose a detector built for the second imbalance. The commit message and changed lines are encoded by a language model pre-trained on code. That representation then acts as a query: it looks into the program's dependence graph around the change and retrieves a small number of relevant nodes. This is cross-modal attention, where one input stream decides which parts of another stream to read. A sparsity-inducing, gated attention step selects only a few nodes, so the rare positive examples send their gradient to the few statements that matter rather than to a blurred average. The claim is falsifiable. The advantage of sparse cross-modal selection over flat or pooled fusion should be small on balanced data and grow as positives become rarer.

Core Claim

Our central hypothesis is an interaction effect. Structure-aware models already exist that fuse code with syntax trees or graphs by concatenation or pooling. Relative to them, a detector whose intent-and-diff query selects a sparse subset of dependence-graph nodes should gain more precision at fixed recall as training prevalence drops from 1:1 toward the natural skew. At balance it should be roughly equal. The mechanism is that sparse selection concentrates each rare positive's learning signal. We test that mechanism directly by checking whether the attention lands on the known faulty statements. We do not claim cross-attention helps defect prediction in general, which the evidence could not support. We claim it helps specifically where the imbalance is severe, and that output-layer reweighting cannot substitute for it.

Relation to Prior Work

Three papers motivate the gap by showing that each ingredient alone is incomplete.

JIT-BiCC pre-trains a bi-modal encoder on commit messages and added and deleted lines. It uses an alignment objective, Replaced Message Identification, in which the model must tell whether a commit message matches its diff. It then concatenates the encoder output with 14 expert metrics in a small feed-forward network. This is the closest existing system and our strongest baseline. It uses no program structure, fuses by concatenation, and trains with plain cross-entropy.

UniXcoder pre-trains over comments, code, and flattened syntax trees by concatenating them into one sequence for self-attention. This inflates input length by about 70%, and the model was never evaluated on imbalanced defect tasks.

Devign combines syntax, control-flow, and data-flow graphs in a gated graph network with static word embeddings. It found that a convolutional selection step beat flat graph pooling by 4.66% accuracy. We read that result as early evidence that selection matters, and we push it further with a query conditioned on developer intent.

Two lines of work constrain the design. DPCAG handles imbalance by propagating defectiveness across class-level dependency graphs. Its node features are static metrics, so it addresses between-sample scarcity, not selection within a sample. He et al. show that detectors trained on balanced synthetic bugs collapse to 3.4% precision on naturally skewed real code. They recover to 64.8% on variable-misuse bugs by first warming up on synthetic data and then adapting while keeping the real skew. Alrasheed et al. show that uncurated synthetic data (all of Draper) hurts more than it helps. From these results we take two rules: synthetic data enters only as a filtered warm-up, and final adaptation and evaluation keep natural prevalence.

The fusion method itself comes from gated cross-attention in drug–target interaction (An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention). There, a learned gate $g$ mixes the cross-attended context with the original features so that neither stream overwhelms the other. We reuse the gate and add sparse attention weights. We found no cross-modal attention work in the file aimed at imbalanced defect prediction.

Research Program

The detector. The query side is a code-pre-trained encoder, initialized from BiCC-BERT or UniXcoder, applied to the commit message and the diff. The key and value side is a program-dependence slice of the changed methods: syntax-tree statement nodes linked by control-flow and data-flow edges, cut to a fixed number of hops from the changed lines. Each node's features come from the same encoder's token embeddings, refined by two rounds of graph message passing. A few learned query vectors, conditioned on the pooled intent-and-diff state, attend over the nodes with a sparse normalizer (sparsemax or entmax in place of softmax), so most nodes receive exactly zero weight. The result passes through the gate $g$ and a focal-loss classifier.

This design keeps compute modest. The cost scales with the number of queries times the number of graph nodes, not with the square of a sequence lengthened by flattened syntax trees.

Structure-grounded pre-training. To test whether the domain-specific pre-training stage earns its cost, we add a short continued pre-training phase on unlabeled commit history. It extends JIT-BiCC's replaced-message objective to graphs:

  • Replaced-slice identification: given a message and diff, decide whether a dependence slice belongs to this change.
  • Masked node recovery: recover masked graph nodes through the cross-attention path.

Two-phase training, following He et al. First, a warm-up on synthetic bugs, filtered to the most realistic 25% by nearest-neighbour similarity to real bugs as in Alrasheed et al. Second, adaptation on real labels at natural prevalence.

Baselines

Naive reference points. The first is the majority-class predictor, which shows how inflated accuracy is under skew. The second is a logistic model on the 14 expert commit metrics. Both anchor MCC and precision-at-recall.

JIT-BiCC, the strongest existing method on commit-level prediction. It is the only system in the file that already combines domain pre-training, developer intent, and defect labels. It reports F1 0.478 and AUC 0.887 on JIT-Defects4J (JIT-BiCC). We will rerun it with its released pre-training recipe on our cross-project splits. We will also run a variant trained with focal loss, which separates their architecture from our loss choice. Beating it means higher F1 and higher precision at 50% recall on the paper's split. On our reduced-prevalence splits it also means a gap that widens as prevalence falls.

UniXcoder with flattened syntax trees, fine-tuned with focal loss and sample weighting. This is the strongest "structure by concatenation" competitor. All methods get the same encoder size, the same number of training steps, and the same hyperparameter search budget of about 20 trials per method. Because flattening lengthens inputs, UniXcoder also gets the same wall-clock budget at its own sequence length.

Devign-style graph network concatenated with the language-model embedding. This is the fusion-by-pooling competitor. It uses the same graph slice and node features as our model, but pools the nodes instead of selecting them.

WizardCoder numbers for the function-level vulnerability track. The reported ROC AUC is 0.86, rising to 0.878 with focal loss and sample weighting on the 1:34 set (Finetuning Large Language Models for Vulnerability Detection). That split was not partitioned by project. We will quote the number for context but compare against a CodeBERT-scale model retrained on our project-disjoint split. We will not claim to beat the 7B-parameter model unless we run it.

Ablations isolating the key ingredient. We will run three:

  • Dense attention: softmax instead of the sparse normalizer.
  • No intent query: learned static queries, not conditioned on message and diff.
  • Token-only sparse attention pooling: the same sparse gated pooling applied over code tokens, with no graph.

Why not just…?

Why not just fine-tune UniXcoder with flattened syntax trees and focal loss? This is the most serious threat. If the structure is present in the input, self-attention might learn the selection by itself. Our prediction is that it matches us at balance and falls behind as prevalence drops, because dense attention over a sequence about 70% longer spreads each rare positive's gradient across more positions. The prevalence sweep settles this. If the two curves stay parallel, the interaction claim is false. We would then report structure-by-concatenation as sufficient.

Why not just concatenate a graph embedding with the language-model embedding? Pooled fusion is exactly what the within-sample imbalance should punish. Devign's own ablation shows that flat pooling loses 4.66% accuracy against a selection module, and JIT-BiCC's concatenation leaves structure out entirely. The Devign-plus-encoder baseline uses identical inputs, so any gap comes from selection versus pooling.

Why not just move the decision threshold or adjust logits for the class prior? Thresholding fixes miscalibrated operating points, but it cannot change how the model ranks examples. Shestov et al.'s small ROC AUC gain from reweighting points to a ranking problem, not a threshold problem. We tune thresholds on validation data for every method and report threshold-free precision–recall AUC. Any remaining gap therefore has to come from the representation.

Why not just do sparse attention pooling over tokens, with no graph? This is a real threat to the "cross-modal" part of the claim. If sparse selection over diff tokens captures most of the gain, the graph is unnecessary and the contribution shrinks to "sparse pooling helps under imbalance." We include this as an ablation and will state the result either way.

Proposed Experiments

Commit-level prediction. The first benchmark is JIT-Defects4J, 27,391 commits from 21 Java projects. We evaluate on the original split, for comparability with JIT-BiCC's 0.478 F1, and on project-disjoint splits. The file does not report this benchmark's natural buggy-to-clean ratio. We will therefore construct training sets at controlled prevalences of 1:1, 1:10, 1:34, and the lowest ratio the data supports, by subsampling positives. Test sets always stay at natural prevalence.

Function-level vulnerabilities. The second benchmark is Devign's C functions. The file gives no absolute Devign number, only relative gains (no reported result found on Lacuna for absolute Devign accuracy or F1). We add a Draper warm-up filtered as in Alrasheed et al.

Metrics. We report MCC, precision–recall AUC, precision at 50% recall, and F1, with ROC AUC for comparison with prior numbers. All results use five seeds with bootstrap confidence intervals.

Decisive check 1: the interaction test. We regress the gap in precision–recall AUC between our model and the UniXcoder and pooled-fusion baselines on log prevalence. A significantly negative slope supports the claim, meaning the gap grows as positives get rarer. A flat slope refutes it.

Decisive check 2: the mechanism test. He et al.'s synthetic Python bugs have known locations, so we can measure the share of attention mass that lands on the faulty statement at each prevalence. If sparse attention localizes well while dense attention spreads out as positives become rare, the mechanism is supported. If the accuracy gain appears without localization, the gain has some other cause, and we will say so.

Decisive check 3: the pre-training test. We remove the structure-grounded pre-training phase. If this costs little at low prevalence, the domain pre-training stage is not justified, and we will recommend starting directly from an off-the-shelf code encoder.