Slotted Code Tokens: Binding Syntax and Data Flow to Every Subword

Code models read programs as flat strings of subword pieces, so a model sees x in x = f(y) and x in return x as the same symbol. It has to rediscover, from raw text, which occurrence defines a value, which one reads it, and how deeply each sits inside nested blocks. Music modeling faced the same kind of problem and solved it with compound tokens: single sequence steps that carry several attributes at once, such as pitch, duration and velocity, each looked up in its own embedding table and summed into one vector. We propose to give each code subword the same treatment. Every subword keeps its position in the sequence but carries parallel "slots" for its syntax-tree node type, its nesting depth, its data-flow role (definition, use, parameter, return value), and an identity for the variable it names. The slots are never added as extra tokens.

We also want to know whether this helps close the gap between natural-language queries and code, not only benchmark accuracy on code-only tasks. Current code search models match docstring wording well and real developer questions poorly. Our hypothesis is that this happens because the models align language with identifier names instead of with what the program does. Slots make that testable. During training we can mask the identifier content of a subword while keeping its structural slots, which forces the query–code alignment to rest on structure. The question is whether this narrows the drop between docstring-matching and real-query retrieval, and whether it helps retrieval of direction-sensitive queries such as "int to string" versus "string to int".

Core Claim

Structural information about code can be bound into each token as extra embedding slots, costing no sequence positions. Once it is bound this way, a slot-level masking objective can make the model's representation of code depend less on identifier names. We claim two things. First, slotted tokens match or beat appended-graph-node models such as GraphCodeBERT at equal context length while using fewer positions. Second, slot masking, rather than the slots alone, is what reduces lexical over-reliance. It should do so measurably on real-query code search, under identifier renaming, and on de-duplicated vulnerability data.

The closest prior work is the following. GraphCodeBERT appends data-flow variable nodes to the token sequence and links them back to code tokens through a masked attention pattern. MV-PTM injects syntax-tree, control-flow and data-flow adjacency matrices into attention heads. AST-T5 keeps flat subwords and uses the syntax tree only to decide segmentation and which spans to mask. All three either spend sequence positions on structure, spend attention memory on it, or leave it outside the representation entirely. None of them binds structure into the per-token input, and none uses structure to control what the model is allowed to see of identifiers.

Mechanism

At position $t$, the input vector is

$$ e_t = E_{\text{lex}}[w_t] + \sum_{k=1}^{K} W_k\, E_k[a_{t,k}], $$

where $w_t$ is the ordinary subword ID and $E_{\text{lex}}$ is the pretrained subword embedding table. Each $a_{t,k}$ is the value of structural slot $k$: tree leaf type, parent type, depth bucket, data-flow role, a within-function variable identity index, and a bucketed offset to the most recent definition of that variable. $E_k$ is the embedding table for slot $k$, and $W_k$ is a projection initialized to zero. The zero initialization means training starts exactly from a pretrained checkpoint such as CodeBERT or GraphCodeBERT.

This design answers the concern raised by MuPT, which found that general-purpose language models fit flat text notations better than multi-slot tuples. Here the flat subword stream is untouched. The slots are additive side-channels, so they cannot disrupt what the model already learned from flat text. Every slot also has a [none] value. Tokens the parser cannot analyze, such as incomplete snippets or unresolved library calls (the case GraphCodeBERT flags with tf.io.read_file), simply receive [none].

The design borrows from music in two ways. The parallel-embedding idea comes from Compound Word Transformer and MusicBERT. MusicBERT also showed that random masking leaks information through correlated neighbors and fixed this by masking whole bars. We adapt that lesson through two slot-level corruptions.

The first is lexical-slot masking. We hide the subword identity of every occurrence of a chosen variable while keeping all its structural slots. This prevents the model from recovering the name by copying it from another occurrence, which is the code analogue of MusicBERT's intra-bar leakage. The model must then predict the name, or match the query, from structure and surrounding context.

The second is structure-slot dropout. We randomly set structural slots to [none] and ask the model to predict their values. This trains the model to infer structure when the parser fails.

For retrieval, these corruptions are also applied to the code side during contrastive query–code training.

Research Program

We will build the slotting pipeline on Tree-sitter for Python, Java and Go. We then continue pretraining from CodeBERT on the CodeSearchNet corpus under several variants. The pretraining objectives are standard masked language modeling, lexical-slot masking and structure-slot prediction. After pretraining, each variant is fine-tuned for code search, clone detection and vulnerability detection. All variants share the same step budget and the same maximum length of 512 positions.

The analysis has three parts. First, we measure how many positions GraphCodeBERT spends on appended data-flow nodes per function. We found no reported figure for this, so we measure it ourselves, and it tells us how much extra code the slotted model fits in the same 512-position window. Second, we measure lexical reliance: how much each model's accuracy falls under consistent identifier renaming. Third, we analyze the query–code gap: the difference between docstring-proxy retrieval and real-query retrieval, which the CodeSearchNet challenge identified as the central failure.

Baselines

The naive reference point is CodeBERT continued-pretrained on the same corpus for the same number of steps, with no slots. For this baseline, GraphCodeBERT reports a code-search MRR of 0.693 on filtered CodeSearchNet. Our rerun uses identical data and steps, so any gain the slotted models show is not an artifact of extra training.

The strongest existing method on the central question is GraphCodeBERT, which appends data-flow nodes and uses graph-guided attention. It reports MRR 0.713 on filtered CodeSearchNet, F1 0.971 on BigCloneBench, and BLEU 80.58 on Java-to-C# translation. It is the right competitor because it injects the same kind of information we do, only by a different route. We run it from its released recipe on our splits, with the same fine-tuning budget and the same hyperparameter search size as our models. Beating it means exceeding 0.713 MRR at equal context length, or matching it while fitting measurably more of each function into the window.

For vulnerability detection, the strongest cited competitor is MV-PTM. It reports F1 0.6843 on FFmpeg and 0.7049 on QEMU, and a 3.36% average F1 gain over GraphCodeBERT. We also evaluate on the de-duplicated dataset from Deep Learning Based Vulnerability Detection: Are We There Yet?. That paper reports only relative gains for its graph model, REVEAL (up to 33.57% in precision and 128.38% in recall), so we found no absolute F1 on Lacuna for it. We will rerun REVEAL ourselves.

For real-query search, the reference numbers come from CodeSearchNet Challenge. A neural bag-of-words model reaches NDCG 0.574, self-attention 0.493, and ElasticSearch 0.337. Beating the bag-of-words figure is the bar. A model that wins on proxy MRR but loses to it on NDCG has not closed the gap.

Two ablations isolate the key ingredient. The first uses slots without the slot-level masking objectives, trained with plain masked language modeling. The second is a flat model trained with identifier anonymization: variable names are replaced by canonical placeholders, with no slots. If the first ablation matches the full model, the contribution is the input representation alone. If the second matches it, slots are unnecessary.

Why not just…?

Why not just give GraphCodeBERT a longer context window? Longer windows address truncation but not the other issue. GraphCodeBERT still needs dedicated masked attention and a node-alignment objective to reconnect each appended variable node to its source tokens, and appended nodes compete with code for every position. Our position-count measurement and a comparison at 512 and 1024 positions will show whether slotting's advantage holds or disappears as the window grows. If it disappears, the efficiency part of our claim is limited to short windows, and we will say so.

Why not just anonymize identifiers in a flat model? This is the most serious threat, because it targets the same lexical-memorization failure that Chakraborty et al. document: flat models lose more than 50% of their accuracy on de-duplicated data. It costs almost nothing to implement. However, anonymization discards names that carry real meaning for natural-language queries. It should therefore hurt real-query search even if it helps under renaming. Slot masking hides names only during training and keeps them at test time. The anonymization baseline decides this directly. If it matches our full model on both renaming robustness and challenge NDCG, the slot mechanism is unnecessary for the retrieval claim.

Why not just add syntax-type embeddings and stop there? The slot-only ablation is exactly this, and it may deliver most of the gain on code-only tasks. Our specific claim is that the query–code gap narrows only when structure is used to decide what the model sees during alignment. If the slot-only ablation closes the proxy-to-real gap as much as the full model, we drop that claim.

Why not a graph neural network over code property graphs, as in REVEAL? Graph networks lack pretrained language knowledge, which matters when one side of the task is a natural-language query. GraphCodeBERT already reports outperforming graph-network models on clone detection. REVEAL remains a strong competitor on vulnerability detection, and we include it as a baseline there.

Proposed Experiments

Code search uses filtered CodeSearchNet (MRR) and the 99-query CodeSearchNet challenge (NDCG). We also build a new direction probe from CodeSearchNet functions: pairs of converse functions, such as serializers and parsers or int-to-string and string-to-int conversions, each with a mirrored query pair. The metric is pairwise accuracy. Clone detection uses BigCloneBench (F1). Vulnerability detection uses FFmpeg, QEMU and the REVEAL data (F1). Robustness is measured as the fraction of each metric retained under consistent identifier renaming, and under random removal of structural information at test time to simulate parser failure.

Four results would decide the proposal. The first is whether the slotted model reaches at least GraphCodeBERT's 0.713 MRR at 512 positions while spending no positions on graph nodes. The second is whether the full model shows a smaller proxy-to-real drop than GraphCodeBERT and beats the bag-of-words baseline's 0.574 NDCG. Our expectation is that this effect appears only with lexical-slot masking, not with slots alone. The third is whether the full model retains more accuracy under renaming than both the flat and the anonymized baselines. The fourth is whether performance degrades smoothly, rather than collapsing, as structural slots are removed. If the gains appear only on proxy MRR and not on challenge NDCG, the direction probe or renaming, the cross-modal claim is falsified. What would remain is a cheaper way to inject structure than appending graph nodes.