Code models are trained and scored on whether their programs pass unit tests. As a result they write code that is correct but slow: nested loops where a hash map would do, full 2D tables where a rolling buffer would do. The tools that currently fix this all run after generation. They profile the output and re-prompt, or they sample many candidates and pick the fastest. Each of these makes the code generator itself slower to serve. This proposal asks whether a change to the generator's shape can close part of that gap in a single pass. The change is to put almost all layers in an encoder, which reads the input program all at once, and very few in the decoder, which writes the output one token at a time.
The idea is narrow on purpose. We study program-to-program rewriting: optimizing a slow program, or translating a program while making it idiomatic and fast. In this setting the algorithmic decision ("this quadratic scan should become a sort plus a sweep") depends mostly on understanding the input. Writing the output is then largely structured copying with local edits. Machine translation has the same structure, and there a deep encoder paired with a one-layer decoder matches symmetric models at much lower latency. We test whether that result carries over to code rewriting. We also test whether it fails in the specific way code would predict: a weak decoder might copy the source too literally, which is exactly the failure mode behind slow translated code.
Core Claim
Hold total parameters and training data fixed. Then a sequence-to-sequence code model with a deep encoder and a one- or two-layer decoder can do three things. It can match a symmetric encoder-decoder and a decoder-only model of the same size on the correctness and runtime of rewritten programs. It can do so in a single forward pass. And it can generate tokens roughly twice as fast. The architecture does not create efficiency on its own; the training targets do. What it does is make an efficiency-aware rewriter cheap enough to remove the inference cost that multi-stage pipelines pay.
The claim is falsifiable at a specific point. On rewrites that require a change of algorithm, as opposed to local idiom fixes, shallow decoders may revert to literal, structure-preserving output. If so, decoder depth is doing algorithmic work and the claim fails.
Grounding in Prior Work
The motivating evidence comes from the efficiency benchmarks. EffiBench profiles 1,000 LeetCode-style tasks against human solutions and finds GPT-4 code 3.12 times slower on average. BEST scores time and memory jointly with a metric called dual@k and finds memory to be the larger blind spot. BigO(Bench) shows that even reasoning models rarely hit a requested complexity class. At production scale, a study of AI-generated C++ at Google measures a 5% compute and 8% memory tax. All of these establish the gap; none touches the architecture of the generator.
The mitigation methods supply our training-signal ideas and our main competitors. EffiLearner injects line-level profiler output into repeated prompts. PerfCodeGen feeds back the slowest test case. Code-Optimise applies preference tuning to self-generated fast/slow pairs. ECCO shows that unguided refinement breaks programs and that execution feedback or fine-tuning on human optimization trajectories helps. The closest work is SwiftTrans. It attacks literal, inefficient translation by sampling several candidates and ranking them with a pairwise selector. Its authors acknowledge that this adds inference overhead. Every one of these methods keeps a decoder-only generator and pays extra test-time compute. We keep their execution-verified signal but move it into training, then ask which generator shape best absorbs it.
The method comes from machine translation. Deep Encoder, Shallow Decoder shows that a 12-1 autoregressive model trained with knowledge distillation matches 6-6 quality while a 6-1 model runs 2.6 times faster. Knowledge distillation here means training on a stronger teacher's outputs. Deep Architectures for NMT finds that encoder depth helps downstream quality while decoder depth mostly lowers training loss, and that layer normalization is needed to train deep stacks stably. RetroMAE shows that a deliberately weak one-layer decoder can force a 12-layer encoder to build richer representations.
Prior work also limits what we can claim. The machine-translation result relies on distillation and on output that tracks the source closely. Code rewrites that change the algorithm break that monotone correspondence, so we cannot assume transfer; we test it. Existing encoder-decoder code models, CodeT5 and CodeT5+, use symmetric depth and ignore runtime. They are our starting checkpoints, not our competitors.
Research Program
We build three students with matched parameter counts, all starting from a CodeT5+ checkpoint. The first is the symmetric original, for example 12 encoder and 12 decoder layers. The second is a deep-encoder variant, roughly 22-2. We remove decoder layers and spend their parameters on extra encoder layers initialized by duplicating existing ones, then follow the layer-normalization practice from the deep-NMT work. The third is a 23-1 variant, which also drops the feed-forward block in the single decoder layer, as the DESD paper did for a further speedup. A fourth student is a decoder-only model of equal size. It isolates "encoder-decoder with shallow decoder" from "any small model". All four receive identical continued pretraining on the same span-corruption token budget, so that differences are not explained by unequal adaptation.
Training targets come from an execution-verified teacher pipeline. A large decoder-only model samples several rewrites of each source program from the ECCO training split. Optionally it also refines them with SwiftTrans-style and EffiLearner-style feedback. We keep only rewrites that pass all tests, including stress inputs in the style of COFFE, and among those we keep the fastest. This pipeline is expensive, but it runs once, offline. Every student is fine-tuned on the same resulting pairs with the same tuning budget.
The analysis has two parts. The first is a decoder-depth sweep at fixed total size. It plots output quality (correctness and speedup) against the model's own latency per generated token. The second splits each test set by whether the reference rewrite changes complexity class, which we estimate empirically with BigO(Bench)-style curve fitting. On each subset we measure literalness: the abstract-syntax-tree edit distance between source and output. This shows whether shallow decoders fail on algorithm changes in particular. As a mechanistic check, we probe each encoder's final states for the complexity class of the reference target. If deep-encoder models encode this decision before the first output token, the probe should be more accurate for them than for symmetric models.
Baselines
The do-nothing reference returns the input program unchanged. It has perfect correctness and a speedup of 1.0 by construction. Any rewriter that loses correctness without beating this speedup is worse than useless.
The strongest existing method is SwiftTrans, from Bridging Functional Correctness and Runtime Efficiency Gaps in LLM-Based Code Translation. It is a fine-tuned Qwen2.5-7B with multi-perspective candidate generation and pairwise diff-based selection. On SwiftBench it reports 90.2% Computational Accuracy and a 292 ms average runtime of generated code. GPT-5 gets 86.4% and 528 ms. We compare against these published numbers on the identical SwiftBench split. We also rerun SwiftTrans where possible to measure its wall-clock serving cost per program. Our student counts as beating it if it comes within two points of 90.2% accuracy, at or below 292 ms output runtime, at a fraction of the serving cost. Matching it outright in a single pass would be a strong result.
On ECCO the reference points are ECCO's own. Exec-Refine keeps 54.6% correctness on CodeLlama, and trajectory fine-tuning lifts CodeLlama-7B from about 43% to over 70%. We fine-tune all students on the same ECCO training pairs and report correctness and speedup on its test split. On EffiBench we turn generation into rewriting: each student optimizes the naive solutions produced by a fixed base model. We compare against EffiLearner, which reports StarCoder2-15B's execution time falling from 0.93 s to 0.12 s. Because EffiLearner runs several profiling rounds, we report output runtime and total compute side by side. For multi-language translation on EffiBench-X we found no reported result on Lacuna, so the numbers there are only relative among our students.
Two ablations isolate the key ingredient. The equal-size decoder-only student trained on the same distilled data separates architecture from data. The deep-encoder student trained on raw ECCO human pairs without teacher distillation tests whether, as in machine translation, distillation is what makes the shallow decoder viable.
Why not just…?
Why not just distill the SwiftTrans pipeline into a small decoder-only model? This is the most serious threat, and the proposal is built around it. The equal-size decoder-only student receives exactly this treatment. If it matches the deep-encoder student on correctness and runtime, then the efficiency gain comes from the distilled data. What remains for the architecture is a latency argument: a decoder-only model runs every layer for every output token, while a 22-2 model runs two. We would then report the result honestly as a serving-cost result, not a quality result.
Why not keep a large decoder-only model and use speculative decoding to cut latency? Speculative decoding means a small draft model proposes tokens and the large model verifies them. It preserves the large model's outputs, so it inherits that model's inefficient-code bias unless the model is also fine-tuned, and it still runs the full model at verification. We include a speculatively decoded fine-tuned teacher on the latency axis of the Pareto plot. If it dominates our students at equal wall-clock time, the deep-encoder design loses on the serving argument.
Why not just prompt for efficient code or use preference tuning? The file's evidence says these are weak. SwiftTrans reports that "write efficient code" prompts introduce bugs. ECCO's unguided Self-Refine drops correctness to 15.8%. Code-Optimise's preference tuning gives only 3–6% runtime reductions against a roughly threefold gap reported by EffiBench. We still run a prompted baseline and a preference-tuned version of the symmetric student as cheap references.
Decisive Checks
Three results would settle the question. First, on the ECCO and SwiftBench subsets that require a change of complexity class, does correctness of the 23-1 student fall sharply relative to the 12-12 student while literalness rises? If so, decoder depth carries algorithmic decisions and the claim fails for those rewrites. Second, does the deep-encoder student beat the equal-size decoder-only student on dual@k from BEST, given that memory is the weakest axis? Third, does the complexity-class probe read the target's class from deep encoders more accurately than from symmetric ones? A positive answer to the last two and a negative answer to the first would support the claim that the efficiency decision can live in the encoder, so the expensive part of efficiency-aware rewriting can be paid once, at training time.
References
- EffiBench: Benchmarking the Efficiency of Automatically Generated Code
- BEST: Benchmarking Efficiency in Space and Time for LLM-Generated Code
- BigO(Bench) - Can LLMs Generate Code with Controlled Time and Space Complexity?
- Characterizing the Quality Profile of AI-Generated C++ in Production
- EffiLearner: Enhancing Efficiency of Generated Code via Self-Optimization
- PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedback
- Code-Optimise: Self-Generated Preference Data for Correctness and Efficiency
- ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness?
- Bridging Functional Correctness and Runtime Efficiency Gaps in LLM-Based Code Translation
- Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
- Deep architectures for Neural Machine Translation.
- RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder
- CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
- CodeT5+: Open Code Large Language Models for Code Understanding and Generation
- COFFE: A Code Efficiency Benchmark for Code Generation
- EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code