Code models read a program the way it was typed, left to right. Natural-language descriptions of that program usually follow a different order: "return the largest value in a list after removing nulls" names the goal first and the filtering step second, while the code does the reverse and wraps both in boilerplate. This proposal tests a simple idea. Before a code encoder sees a function, a small learned module rewrites its input in two constrained ways. It reorders whole syntactic units, such as statements and argument blocks, so they follow the order in which a human would describe them, and it drops units that carry no information about the description. Both operations only permute or delete pieces of the original syntax tree, so the result can never contain an invented variable or a broken scope. That is the failure that sank neural pseudocode generators in earlier work.
The opportunity matters because the strongest evidence in the literature about closing the gap between natural language and code points at exactly these two operations, but it got them through hand-written rules. Replacing raw code with rule-generated pseudocode raised code-search mean reciprocal rank (MRR) on NAPS+ from 0.0880 to 0.2485, while neural code-to-pseudocode generators hallucinated and failed (Bridging Code-Text Representation Gap using Explanation). We ask whether a learned "reorder and prune" step can recover that gain without per-language rules and without hallucination, on standard benchmarks where structure-aware encoders are already strong.
Core Claim
Part of the code–text representation gap is an ordering and redundancy problem, not only a vocabulary problem. A code-only module that permutes syntax subtrees into description order and gates out description-irrelevant subtrees should improve code search and summarization over the same encoder fed code in its original order. It should also beat any fixed heuristic reordering. Two ideas from outside code supply the mechanism. Source-side pre-ordering from machine translation makes a hard non-monotonic alignment roughly monotonic before the main model runs. The information bottleneck, an objective that keeps only the parts of an input that predict the target, is used here as a learned, text-guided pruning rule over syntax subtrees. What would be new is learning a subtree-constrained reordering and bottleneck gate for code, supervised by paired text during training and needing only the code at inference.
How it relates to prior work
Three structure-aware code encoders motivate the work and serve as the backbone. CodeBERT treats code as a flat token string. GraphCodeBERT adds a sparse data-flow graph through attention masks and raises CodeSearchNet MRR from 0.693 to 0.713. AST-T5 uses the parse tree to choose segmentation boundaries and masked spans. All three keep the code's original token order, and none removes content. We keep their encoders and change only what they are fed. Evidence that flat code models latch onto surface tokens (Deep Learning Based Vulnerability Detection: Are We There Yet?) motivates the pruning half.
The reordering half borrows its method from translation. Dependency Treelet Translation learns to predict where each modifier goes relative to its head. CURSOR searches over source permutations under a hard constraint that a subtree moves as a unit, and reports +15.3 attachment-score points on Hindi. We reuse the subtree constraint and the idea of predicting relative order. Our ordering target comes from code–text alignment rather than a target-language part-of-speech model, and we replace genetic search with a learned scoring function that sorts subtrees in a single pass.
The pruning half borrows from two bottleneck methods. Graph Information Bottleneck gates graph structure with Bernoulli or Gumbel-Softmax sampling and has only been tested on citation graphs. IBMA uses one modality to guide compression of another, with a prototype-based bound whose cost grows linearly in batch size, and has only been tested on audiovisual and medical data. We apply GIB-style subtree gates with IBMA-style text guidance to syntax trees.
One constraint shapes the design. The heuristic body-stripping studied in Commenting Higher-level Code Unit cut tokens by 74.65% but lost about 23% of summary quality. Pruning must therefore be learned and selective, not wholesale deletion of logic.
Research Program
We parse each function with Tree-sitter and treat statement-level and argument-level subtrees as movable units. Every unit has a parent, and children travel with their parent, as in CURSOR. A light scoring network reads each unit's pooled encoder states and outputs two things: a sort key, which decides the unit's position among its siblings, and a keep probability. Positions are learned with a differentiable sorting relaxation. Keep decisions are sampled with a Gumbel-Softmax gate. The rewritten sequence goes into a GraphCodeBERT-initialized encoder, with each surviving token's data-flow edges preserved.
Training uses paired descriptions in two ways. First, a frozen cross-encoder aligns description words to code units. Sorting the units by the position of their best-aligned word gives a target order, and the scorer is trained to reproduce it. This is the code analog of predicting a modifier's target position. Second, the gates are trained under a bottleneck objective:
$$\mathcal{L} = \mathcal{L}_{\text{task}}(t, f(\pi_\phi(m_\psi \odot u(x)))) + \beta\, \mathbb{E}\!\left[\tfrac{|m_\psi|}{|u(x)|}\right] + \lambda\, \mathcal{L}_{\text{order}}.$$
Here $x$ is the code and $u(x)$ its sequence of units. $m_\psi$ is the binary keep mask and $\pi_\phi$ the learned permutation. $f$ is the encoder and $t$ the paired text. $\mathcal{L}_{\text{task}}$ is the contrastive loss for search or the decoder loss for summarization. The second term penalizes the expected kept fraction, a crude stand-in for $I(X;Z)$ that we will compare against IBMA's prototype bound. $\mathcal{L}_{\text{order}}$ is the ranking loss toward the alignment-derived order. At test time the description is not used. Indexed code is rewritten once, offline, so search latency is unchanged.
Baselines
The do-nothing reference is the same GraphCodeBERT-initialized encoder fine-tuned on code in its original order, with the same data, steps, and hyperparameter budget. Every other system is compared against it.
The strongest existing method on code search is GraphCodeBERT itself. Its paper reports overall CodeSearchNet MRR 0.713 (GraphCodeBERT). We will rerun it in our pipeline rather than rely on the reported figure, because the CodeBERT paper reports a different CodeSearchNet setup (macro MRR 0.7603) that is not directly comparable. On NAPS+ the strongest reported number is rule-based pseudocode replacement at 0.2485 MRR against a 0.0880 no-explanation baseline (Bridging Code-Text Representation Gap using Explanation). We will reimplement that pipeline where its rules apply. For code-to-text summarization on CodeXGLUE and CCSD, the file gives only relative gains: HGNN beats Rencos by 1.42 BLEU-4 on CCSD (Retrieval-Augmented Generation for Code Summarization via Hybrid GNN). No absolute reported result was found on Lacuna for these, so we will report rerun baselines only.
Three ablations isolate the key ingredient. In the first, units are reordered but nothing is dropped (gates off). In the second, units are pruned but keep their original order, which amounts to GIB-style gating on syntax trees. In the third, a variational bottleneck is applied to token embeddings with no structure. A fourth control applies a random subtree-respecting permutation. It tests whether any gain is really an augmentation effect rather than an alignment effect.
Beating the field means three things. On CodeSearchNet, the full model must exceed the rerun GraphCodeBERT by more than the seed-to-seed spread across five seeds, and must also beat every heuristic ordering below. On NAPS+, it must reach or exceed rule-based replacement without hand-written rules.
Why not just…?
Why not just use a fixed heuristic ordering, such as a data-flow topological sort or putting the return statement first? This is the most serious threat. If a deterministic order captures most of "description order," the learned scorer is unnecessary. We therefore include three heuristic orderings as baselines: data-flow topological, return-first, and reversed statements. If the best of them matches the learned order within noise, we will report that and recommend the heuristic. That would still be a new, useful finding that code order matters.
Why not just rely on attention, which can relate any two positions? The translation evidence (CURSOR, Dependency Treelet Translation) says that removing reordering burden helps even strong models. That evidence comes from parsers and decoders, though, not dual encoders that output a single vector. The test is direct. We measure, per example, how many alignment links cross in the original order. The proposal predicts that gains concentrate in high-crossing examples and vanish in already-monotonic ones. If gains are flat across crossing levels, the ordering story is wrong.
Why not just use rule-based pseudocode or body stripping? Rules worked on NAPS+ but must be rewritten for each grammar. Wholesale stripping lost about 23% of summary quality (Commenting Higher-level Code Unit). The head-to-head on NAPS+ settles whether learning matches the rules. CodeSearchNet's six languages test whether it transfers where no rules exist.
Why not just scale the flat model? AST-T5 at 277M parameters reached 14.0 Pass@1 on HumanEval against 5.2 for a same-size T5 (AST-T5), so structure buys more than parameters at this scale. We hold backbone size fixed and do not claim results beyond it.
Proposed Experiments
The primary benchmark is CodeSearchNet retrieval across all six languages, scored by MRR. NAPS+ is the comparison against rule-based bridging. CodeXGLUE code-to-text and CCSD summarization, scored by BLEU-4, ROUGE-L, and METEOR, test whether monotonic input helps a decoder.
The decisive checks are four. The learned order must beat the best heuristic order. Gains must concentrate in high-crossing examples. A sweep of $\beta$ must trace a curve where moderate pruning helps and heavy pruning hurts. On a copy of CodeSearchNet with identifiers consistently renamed, the pruned model must lose less MRR than the baseline, which would show that pruning removed reliance on surface names in the way Deep Learning Based Vulnerability Detection: Are We There Yet? warns about.
We will also report which unit types the gates drop: logging, null checks, or wrapper expressions. The aim is to check that the bottleneck removes boilerplate rather than logic.
References
- Bridging Code-Text Representation Gap using Explanation
- CodeBERT: A Pre-Trained Model for Programming and Natural Languages
- GraphCodeBERT: Pre-training Code Representations with Data Flow
- AST-T5: Structure-Aware Pretraining for Code Generation and Understanding
- Deep Learning Based Vulnerability Detection: Are We There Yet?
- Dependency Treelet Translation: Syntactically Informed Phrasal SMT
- Cross-Lingual Dependency Parsing by POS-Guided Word Reordering
- Graph Information Bottleneck
- IBMA: Information Bottleneck-Based Multimodal Alignment
- Commenting Higher-level Code Unit: Full Code, Reduced Code, or Hierarchical Code Summarization
- Retrieval-Augmented Generation for Code Summarization via Hybrid GNN