Branch-Inversion Equivariance for Multi-Task Code Models

Programs can be rewritten in many ways without changing what they do. One of the most common rewrites in real code is to negate an if condition and swap the two branches: if (!p) A else B behaves exactly like if (p) B else A. Further rewrites of the same kind include mirroring a comparison (a < b becomes b > a) and De Morgan's laws. A model that reads code as text sees these pairs as different inputs. Vulnerability detectors trained this way are known to latch onto surface patterns instead of program logic, and they lose more than half their performance on realistic, de-duplicated data (Deep Learning Based Vulnerability Detection: Are We There Yet?). The usual fixes are to discard the surface form or to augment the data. Both create a problem in multi-task training, because some tasks genuinely need the surface form. Predicting a masked ! or repairing a < that should be <= depends on exactly the details that a semantic task like "is this function vulnerable?" should ignore.

We propose to treat these rewrites as an exact symmetry with a sign-flip structure and build it into the network. Each feature channel is labelled as either "even" (unchanged by the rewrite) or "odd" (its sign flips under the rewrite). A known theorem gives a precise rule for which nonlinearities are allowed on such channels, and we follow it. One shared backbone can then serve both kinds of task: semantic heads read only the even channels and are exactly invariant to the rewrites, while surface heads also read the odd channels and can tell the forms apart. The bet is that this structural routing removes a source of negative transfer between surface and semantic objectives, which loss reweighting, augmentation, and soft symmetry penalties cannot remove.

Core Claim

A Characterization Theorem for Equivariant Networks with Point-wise Activations proves the following. If a network applies its nonlinearity coordinate by coordinate, then odd activations such as tanh are exactly the ones compatible with signed-permutation symmetry, meaning permutations combined with sign flips. Arbitrary activations such as ReLU or GELU are compatible only with plain permutations. The theorem is usually read as a warning for 3D rotations. We read it as a design rule for code, because branch inversion is a signed permutation.

Consider a branch node with condition encoding $c$ and the messages $t$ and $e$ arriving from its then-branch and else-branch. Inversion maps $(c, t, e) \mapsto (-c, e, t)$. If we rewrite the branch messages as a sum $s = t + e$ and a difference $d = t - e$, inversion simply flips the signs of $c$ and $d$ and leaves $s$ alone. The construction follows the theorem's two maximal cases. Odd activations act on the odd channels $c$ and $d$. Arbitrary activations act on the even channel $s$. The two parities are connected by scalar gating, borrowing the gating construction from G-RepsNet and A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups. In the discrete setting the gate is a product of two odd quantities:

$$u = c \odot d,$$

Here $\odot$ is elementwise multiplication, and $u$ is even. In plain terms, $u$ records "which behaviour happens when the condition holds", and it does not depend on how the condition was written. Only even features ($s$, $u$, $|c|$) travel beyond the branch node. Every branch node therefore enforces its own sign flip locally, and features never need to carry products of many unrelated parities.

The condition encoder is made odd under negation by construction. not x multiplies its encoding by $-1$. The operator or is defined from and as $f_{\text{or}}(x,y) = -f_{\text{and}}(-x,-y)$, which builds De Morgan's laws in exactly while leaving the inner function $f_{\text{and}}$ unconstrained. Comparisons are defined from a single learned function $\ell(a,b)$ for a < b. We set a > b to $\ell(b,a)$, a >= b to $-\ell(a,b)$, and a <= b to $-\ell(b,a)$. Because $\ell$ is not forced to be antisymmetric, a < b and a <= b keep distinct encodings. This matters because the off-by-one distinction is precisely the kind of single-token semantic change that a code model must not erase.

Research Program

The backbone is a small token Transformer that embeds operands and statements. On top of it sits a graph network over the control-flow and data-flow graph, in the style of the composite graphs of Devign. Branch and condition nodes are replaced by the parity-structured updates described above. Ordinary nodes keep their usual updates, since node permutations are already compatible with arbitrary pointwise activations according to Pacini et al.

Training is multi-task, with the three objectives of GraphCodeBERT: masked token prediction, data-flow edge prediction, and token-to-node alignment. A semantic target task is trained jointly with them. The architectural rule is that semantic heads (defect detection, clone detection, execution output) see only even channels, while the masked-token and repair heads see odd channels as well.

For execution tasks, the parity structure fits directly into the soft instruction pointer of IPA-GNN. IPA-GNN models execution as a probability distribution over which statement runs next. If the branch probability is computed with a bias-free logit,

$$p_{\text{then}} = \mathrm{sigmoid}(w^\top c),$$

where $w$ is a learned vector, then inversion sends $p_{\text{then}}$ to $1 - p_{\text{then}}$ while the branches swap. The soft execution trace is therefore exactly unchanged.

Beyond accuracy, the analysis component measures gradient conflict. We record the cosine similarity between the gradients of the masked-token loss and the semantic loss on the shared parameters during training. The hypothesis is that in a standard backbone, the masked-token objective pushes the shared features to encode surface form, and the semantic head then exploits that form. That should appear as conflicting or misaligned gradients. Parity routing should reduce the conflict without any reweighting.

Baselines

The naive reference point is a standard graph model with ordinary GGNN gates and GELU activations, together with a token-only encoder. CodeBERT reports 62.08% accuracy on Devign defect detection (CodeXGLUE). Devign reports 74.11% accuracy on 40 recent real-world vulnerabilities from its own C corpus.

The strongest existing structural competitor is GraphCodeBERT trained with its three pre-training objectives plus the target task, using the fixed unweighted loss sum it reports. It reaches 0.971 F1 on BigCloneBench. No reported result was found on Lacuna for GraphCodeBERT on the Devign or REVEAL vulnerability datasets. REVEAL reports only relative gains (up to 33.57% precision and 128.38% recall over its baselines), so for REVEAL no absolute reported number was found on Lacuna, and we will rerun its CPG+GGNN+triplet pipeline from the public replication package.

For fairness, all graph models are trained from scratch with matched parameter counts. They use the same unlabeled C functions for the auxiliary objectives (Devign plus REVEAL), the same number of steps, and the same hyperparameter search budget. Released GraphCodeBERT checkpoints are reported separately as a reference, not as the controlled comparison.

Three further competitors isolate the key ingredient. The first is REMUL (Relaxed Equivariance via Multitask Learning), run with the same backbone and a GradNorm-balanced auxiliary loss that penalizes differences between semantic-head outputs on a function and on its rewritten twin. The second is the same backbone trained with RotoGrad (RotoGrad: Gradient Homogenization in Multi-Task Learning). This tests whether any gain is really an optimization effect that a gradient-balancing method would also deliver. The third is an ablation that keeps the parity-split architecture but uses GELU on the odd channels. By the theorem, this breaks exactness, so the comparison isolates the role of the activation choice.

Beating the field means two things. First, the parity model must match or exceed the controlled GraphCodeBERT-style model on the original test sets. Second, it must lose nothing on rewrite-perturbed test sets, while every baseline shows a measurable drop there.

Why not just…?

Why not just augment the training data with inverted branches? Augmentation is the most natural competitor, and we will run it with the exact rewrite engine used to build the test perturbations. It gives approximate invariance only on the rewrites it samples. It also gives the masked-token head conflicting targets for semantically identical inputs. The evidence from REVEAL is that data-driven fixes without structural bias still memorize artifacts. The decisive test is a skewed-training experiment in which augmentation covers only depth-1 inversions and testing uses nested inversions. Exact equivariance holds at any depth by construction, so a gap should open there.

Why not just canonicalize the code, rewriting every condition into a normal form, before the model sees it? This is a real threat. For a purely semantic single-task model, canonicalization gives exact invariance for free, and we expect it to tie with our model in that setting. We report that tie honestly. Canonicalization stops working when the same backbone must also predict or repair surface tokens, because the surface form has been erased. The fair version of this alternative feeds canonical code to the semantic heads and raw code to the surface heads through shared weights, at twice the forward cost. If this version matches ours on accuracy and on gradient-conflict measurements, the contribution reduces to a compute saving, and we will say so.

Why not just use the soft equivariance penalty of REMUL? REMUL was shown to work on physics benchmarks with continuous symmetries, where exact equivariant layers are slow. Here the exact construction costs almost nothing: a sign convention and a choice of activation. A soft penalty introduces a weight that must be tuned against the task loss. The REMUL baseline described above measures whether the soft version leaves residual sensitivity to rewrites, and whether it reproduces the reduced gradient conflict.

Proposed Experiments

The first check is exactness. For every test function we enumerate its rewrite orbit up to a fixed depth and confirm that semantic-head outputs are identical up to floating-point error. We also confirm that surface heads still distinguish the members of the orbit.

The second experiment uses the synthetic program execution benchmark of IPA-GNN, where the reported full-execution accuracy is 62.1% (oracle trace model: 66.4%). We train on the original distribution and test on both the original and a randomly branch-inverted copy, across the paper's split of training lengths up to 10 and test lengths up to 100. If the parity IPA-GNN does not lose accuracy on the original set, the equivariance constraint costs nothing there. The inverted set then shows whether the unconstrained IPA-GNN has learned a surface shortcut.

The third experiment covers vulnerability detection on Devign and REVEAL. We report accuracy and F1, and on REVEAL we report precision and recall at its roughly 9% positive rate. Each model is evaluated on the original test split and on a rewrite-perturbed split. We also run an orbit-shift study in which training functions are filtered to one guard style (such as if (!p) return) and test functions use the opposite style.

The fourth experiment checks transfer and the absence of harm. On BigCloneBench, near GraphCodeBERT's 0.971 ceiling, we verify that clone F1 does not regress. We also add a stress set of clone pairs that differ only by inversion rewrites.

The final experiment is the multi-task diagnosis. For each architecture we compare semantic-task accuracy with and without the masked-token auxiliary, and we log the gradient cosine described earlier. Our prediction is specific. The auxiliary objective should help the parity model and hurt or not help the plain and augmented models. Gradient conflict between surface and semantic losses should be measurably lower in the parity model. If neither prediction holds, the claim that equivariant routing reduces negative transfer is wrong, and what remains is a robustness result.