Compiler-Verified Consistency Training for Robust Models of Code

A deep network that reads computer programs is strangely easy to fool. Change a handful of variable names — edits that provably cannot change what a program does, since compilers ignore names — and state-of-the-art programming language models like CodeBERT collapse: CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Models reports attack success rates of 90–99% by altering as few as one to three tokens, cutting summarization BLEU by nearly 50%. The root cause is that these models lean on the natural channel of code — readable identifier names, comments, and style — rather than the formal channel of syntax and data flow that actually determines behavior. Consistency regularization, which trains a model to give the same prediction on two "views" of the same input, is the natural medicine: if two programs are provably equivalent, any prediction difference is pure error. But in code, unlike images, a random "view" is likely to be a broken program. The transformations must themselves be verified to preserve semantics.

This proposal builds a training procedure that closes this loop: a library of semantics-preserving source-code transformations, each candidate checked by a compiler and lightweight static analysis before it is used; a cheap gradient-ranking step that picks the worst-case verified transform, so consistency is enforced where the model is most fragile; and an asymmetric, stop-gradient consistency loss that smooths the decision boundary without collapsing it. The same loss runs on unlabeled code, which matters because labeled code is brutally expensive — Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networks required 600 person-hours to label 59,000 functions, while CodeBERT: A Pre-Trained Model for Programming and Natural Languages shows millions of unlabeled functions sitting in CodeSearchNet.

Core Claim

The claim is that enforcing prediction consistency between a program and a compiler-verified, worst-case semantics-preserving variant of it makes models of code simultaneously (i) robust to renaming and dead-code attacks at a level that plain adversarial training does not reach, (ii) no worse on clean data, and (iii) able to exploit unlabeled repositories for large label-efficiency gains. The critical design choice is where the semantic guarantee comes from. In natural language, no such guarantee exists: Consistency Training with Virtual Adversarial Discrete Perturbation (VAT-D) filters candidate token swaps with a masked language model's probabilities — a statistical guess that, as its authors admit, still yields ungrammatical sentences. In code, "ungrammatical" means the program does not compile, and we can check that exactly. We swap the heuristic filter for a compiler, and we swap the labeled-only setting of prior code defenses for one that also consumes unlabeled functions.

Concretely, for a training program $x$ with label $y$, we generate a candidate set $\mathcal{C}(x)$ of verified variants using identifier renaming, dead-code insertion, and equivalent-statement substitution, all accepted only if the variant compiles and passes a static check that scope, type, and data flow are unchanged. From $\mathcal{C}(x)$ we select the variant the model currently fears most:

$$x^\* \;=\; \arg\max_{x' \in \mathcal{C}(x)} \; \mathcal{D}\big(f_\theta(x),\, f_\theta(x')\big),$$

where $f_\theta$ is the current model and $\mathcal{D}$ is a prediction discrepancy. Training then minimizes the task loss on $x$ plus an asymmetric consistency term:

$$\mathcal{L} \;=\; \mathcal{L}_{\text{CE}}\big(f_\theta(x), y\big) \;+\; \lambda \,\big(1 - \cos\big(z_\theta(x),\, z_\theta(x^\*)\big)\big),$$

where $z_\theta(\cdot)$ is the encoder's pooled representation and the clean branch is detached (stop-gradient), so the transformed branch is pulled toward the clean anchor, never the reverse. Cosine similarity on latents follows the empirical finding in Augmentation-induced Consistency Regularization for Classification that it is more stable than KL divergence, and the anchor-plus-stop-gradient structure follows the collapse-prevention mechanisms validated there and in ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring. On unlabeled functions, the same loss runs with the clean view's sharpened prediction as a pseudo-label, exactly the anchoring trick from ReMixMatch — never before applied to discrete, syntax-checked code.

What Exists and What This Adds

Two prior works sit closest, and both leave the proposed setting open. RoMA: Robust Malware Attribution via Byte-level Adversarial Training with Global Perturbations and Adversarial Consistency Regularization does combine semantic-integrity-constrained perturbations with consistency losses, and its results motivate us: on the AMG18 attribution benchmark it held 80.13% accuracy under a 50-step PGD attack where four-step adversarial training managed only 36.85%, while improving clean accuracy (91.00% vs. 90.06%) and training roughly twice as fast (2,378 minutes). But RoMA operates on raw Windows binaries with a fixed learned pool of byte patterns, purely as a supervised defense. We work on source code, where transformations can be program-aware (renaming a specific variable the attention mechanism fixates on) rather than byte-pattern injections, and we extend the consistency objective to unlabeled code, which a supervised defense like RoMA cannot use. Marvolo: Programmatic Data Augmentation for Deep Malware Detection supplies the verification half — ten formally verified, semantics-preserving binary mutations — but pools the mutants into the training set under plain cross-entropy, with no consistency loss and modest gains (up to 6% on Ember subsets, around 2% on the Brazilian dataset). We reuse its verification philosophy (static checks, discard behavior-altering edits) but put the verified variants inside a consistency objective rather than the data pool. On the attack side, Generating Adversarial Computer Programs using Optimized Obfuscations formalizes functionality-preserving obfuscations as site-selection/site-perturbation optimization and defends with standard min-max adversarial training; its reported ~16-point robustness gain is our strongest competitor, but it enforces no consistency between clean and perturbed programs, uses no unlabeled data, and inherits the instability of min-max training on the "spiky" discrete loss landscape its own authors document. Finally, the entrenchment warning of Consistency Training Can Entrench Misalignment constrains our design: consistency over self-generated pseudo-labels can amplify shortcuts. Our anchor is always the unmodified program, and our transformation library deliberately destroys the natural channel (renames every identifier family), so the shortcut features consistency would entrench are precisely the ones being destroyed.

Research Program

Three components get built. First, a verified transformation engine for Java and C/C++: candidate generators for consistent identifier renaming (in the spirit of Adversarial Examples for Models of Code, whose DAMP attack shows gradient-guided renaming is the highest-leverage natural-channel edit), dead-code insertion (following DIP: Dead code Insertion based Black-box Attack for Programming Language Model, which guarantees 100% compilability), and equivalent-statement substitution — each passed through compilation plus AST and data-flow comparison before acceptance, mirroring Marvolo's static verification pass. Second, the selection-and-training loop: ranking candidates by a one-forward-pass gradient proxy rather than running an expensive inner optimization, because query cost is a documented bottleneck (A New Formulation for Zeroth-Order Optimization of Adversarial EXEmples in Malware Detection emphasizes hundreds of queries per sample in zeroth-order settings, and DIP needed 106.2 queries even for an efficient black-box attack). Third, the semi-supervised extension: pseudo-labeling from the detached clean anchor over unlabeled CodeSearchNet functions, with distribution alignment against class marginals as in ReMixMatch to avoid degenerate collapse.

Analysis accompanies each component: measurements of how the candidate-pool size trades off against verification cost; whether cosine-on-latents versus KL-on-predictions changes stability in the discrete setting; and a mechanistic check that consistency under renaming shifts attention mass from identifier tokens toward structural tokens, directly testing the natural-versus-formal channel hypothesis of CodeAttack.

Proposed Experiments

The supervised robustness experiment fine-tunes CodeBERT and GraphCodeBERT on three tasks from the file: vulnerability detection on the Devign dataset (59k C functions; the Devign paper itself reports 74.11% accuracy on 40 real-world CVEs and a 10.51% average accuracy edge over baselines, giving us calibrated expectations), clone detection on BigCloneBench, and code summarization on CodeSearchNet. We then attack every trained model with CodeAttack (reported 90–99% ASR, near-50% BLEU drops on summarization), DIP (43.9% average ASR at 106.2 queries, with CodeBLEU 0.88 confirming semantic preservation), and DAMP-style renaming. Metrics are clean accuracy/F1/BLEU, attack success rate under each attack family, and robustness under cross-family attacks — testing on a transformation family not used in training — since transferability failures are documented in Evaluating the Robustness of a Production Malware Detection System to Transferable Adversarial Attacks and A Wolf in Sheep's Clothing: Practical Black-box Adversarial Attacks for Evading Learning-based Windows Malware Detection in the Wild. The label-efficiency experiment restricts Devign labels to 40 and 250 total examples (regimes where ReMixMatch reached 84.92% and 93.73% on CIFAR-10) and asks whether anchored consistency over unlabeled functions recovers a useful fraction of full-label performance. The ablation experiment isolates each ingredient: gradient-ranked selection versus random verified variants, compiler verification versus unverified MLM-proposed edits, cosine versus KL, and anchor/stop-gradient on versus off. A final entrenchment audit measures reliance on identifier tokens by evaluating on identifier-anonymized code, guarding against the failure mode Africa and Mani describe.

Baselines

The do-nothing reference is standard fine-tuned CodeBERT and GraphCodeBERT with no robustness training; the documented reference point is severe fragility, with CodeAttack reporting 90–99% attack success and near-50% BLEU reduction on summarization, and we will re-measure these numbers on our splits. The strongest existing defense in the literature is adversarial training with optimized obfuscations from Generating Adversarial Computer Programs using Optimized Obfuscations, which reports a robustness improvement of roughly 16 points on Python program summarization; we re-implement its min-max training on our transformation library so that the comparison isolates the loss shape (consistency versus cross-entropy on worst cases) rather than the perturbation set, and we give it the identical base model, data, tuning budget, and total forward/backward pass budget. The second defense baseline is Marvolo-style empirical risk minimization: verified mutants pooled into the training set with plain cross-entropy, again on our library, so the comparison isolates the consistency loss itself. The third is R-Drop-style dropout consistency, which needs no compiler engineering at all and therefore tests whether domain-specific transformations matter. The key ablation replaces gradient-ranked worst-case selection with uniform random choice among verified candidates; if the two tie, the adversarial selection ingredient is dead weight and we would report that. Beating the adversarial-training baseline means matching its clean accuracy within one point while reducing attack success rate under at least one cross-family attack it cannot resist, at no greater training wall-clock — with the overhead honestly reported, since consistency training is known to run roughly 2.5× slower than standard training in VAT-D's discrete setting.

Why not just…?

Why not just add the verified mutants to the training set? This is the Marvolo recipe, and the evidence says plain pooling leaves most of the gain on the table. On identical augmented data, Augmentation-induced Consistency Regularization for Classification is the wrong file — the correct citation is Augmentation-induced Consistency Regularization for Classification, which shows standard cross-entropy on augmented data reaches 81.71% on CIFAR-10 where adding the consistency loss reaches 93.41%; and Marvolo's own supervised gains on malware were only 2–6%, with its authors warning that mutation pooling risks memorizing mutation artifacts. Our experiments include this baseline head-to-head, so the claim is testable rather than imported from vision.

Why not just use dropout consistency (R-Drop) and skip the compiler engineering? Dropout perturbs the model, not the input, so it can never teach invariance to a structured external shift like renaming every variable. The numbers back this up: R-Drop managed 82.13% on CIFAR-10 where data-level consistency reached 93.41%, and on Google Speech Commands 96.52% versus 97.21% (Augmentation-induced Consistency Regularization for Classification is again mistyped — the correct artifact is Augmentation-induced Consistency Regularization for Classification). We will still run R-Drop as a baseline because it is free, but the mechanism cannot target the natural channel.

Why not just do standard min-max adversarial training? This is the one real threat, and we treat it as the primary baseline rather than arguing around it. The counter-evidence is that RoMA's consistency-based defense more than doubled robustness over multi-step adversarial training under strong attack (80.13% versus 36.85% under PGD-50) while improving clean accuracy and training faster — and Srikant et al. themselves document that the discrete code loss landscape is spiky, which makes the inner maximization of min-max training unreliable. But those results come from binaries and from program summarization respectively, not from our tasks, so our head-to-head experiment under equal budgets is the decisive test; if min-max training matches our robustness on Devign and BigCloneBench, the proposal's remaining claim narrows honestly to label efficiency and clean-accuracy preservation, and we would say so.

Why not just trust a masked language model to propose valid transforms, as VAT-D does? In natural language an occasionally ungrammatical sentence is a minor nuisance; in code an invalid transform silently corrupts the training signal — the model is being asked to be consistent with a program that no longer means the same thing. VAT-D's own authors note that simultaneous token replacement often yields ungrammatical text, while DIP demonstrates that compiler-checked dead-code insertion achieves 100% compilability. The MLM-only variant is exactly our verification ablation, so the cost of the compiler pass will be measured, not assumed away.

Closing Checks

The proposal is falsifiable at three points. If random verified variants match gradient-ranked ones, the selection machinery is unnecessary. If min-max adversarial training matches us under cross-family attacks at equal budget, the consistency loss adds nothing over worst-case supervision in this domain. And if anonymized-input evaluation reveals no shift of attention away from identifier tokens, the natural-channel mechanism story is wrong even if the accuracy numbers hold. Each check is built into the experiment matrix above, and no reported result we rely on extends beyond the file's evidence: for the semi-supervised code setting itself, no reported result found on Lacuna for label-efficient consistency training on unlabeled source code.