When a language model translates a program from one language to another, many tokens in the output should not be translated at all. Variable and function names, string and numeric constants, and API names that exist in both languages should reappear exactly as they were in the source. Today these tokens are regenerated from the model's vocabulary like any other word, and they sometimes come back renamed, rounded, or swapped for a more common name. A compiler often cannot catch this, because the program still compiles. The feedback loops that let a model repair its own translation from compiler errors also rewrite code that was already correct, and their gains stop after about three rounds. We propose to make exact preservation a decoding operation rather than an instruction. At each step the model may copy a whole span from the source program. Copying is allowed only for spans a parser has marked as "must survive" (identifiers and literals), and in later repair rounds also for lines of the previous draft that the compiler did not flag.
The mechanism is a pointer-generator, a decoder that at each step mixes ordinary generation from its vocabulary with copying a token it points to in the input. The literature contains a clear negative result for it: pointer-generators gave no benefit in translation, and on sentences with many shared subwords they were slightly worse (Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That!). Our bet is that this failure has a specific cause that is absent in code. Natural-language cognates are almost identical across languages but not exactly identical, so copying single subword pieces copies the wrong thing. Program identifiers and literals are byte-identical across languages and pass through the same tokenizer on both sides. The copy unit is therefore well defined, provided copying happens at the level of whole spans and is restricted to spans that should be copied.
Core Claim
The claim has three parts. First, an inference-time copy mechanism should measurably reduce failures in model-based code translation, both before any repair round and during repair rounds. The mechanism needs no retraining. It operates on whole parser-delimited spans, and its copy gate opens only when the current output prefix matches one of those spans. Second, the gain should concentrate on failures caused by a source identifier or literal being altered. Third, the same mechanism should give no benefit, or slight harm, on natural-language pairs where copyable units are only near-identical. That third result would reproduce and explain the negative finding above, not contradict it.
The method builds on LargePiG, which turns a decoder-only model into a pointer-generator without training. It reads the model's own attention to the input as the pointer distribution and sets the copy probability from the Jensen-Shannon divergence between intermediate-layer and final-layer vocabulary distributions. LargePiG was tested only on one-shot question and query generation, and it copies single tokens wherever attention points. That is the same lack of selectivity that the low-resource translation study observed, where models copied arbitrary subwords. We keep LargePiG's training-free gate and add two constraints from the code domain: copying is restricted to a typed set of spans, and copying operates on whole spans.
Research Program
Before decoding, a parser extracts an anchor set $S$ from the source program: identifiers, string literals, numeric literals, and qualified API names. Later, for repair rounds, we add every line of the previous draft that lies outside the region the diagnostics point to. At decoding step $t$, let $S_t \subseteq S$ be the anchors whose token sequence is consistent with the current output suffix. This includes anchors not yet started, which count as consistent at any token boundary. The output distribution is
$$P(y_t) = (1-g_t)\,P_{\text{LM}}(y_t) + g_t \sum_{s \in S_t} \alpha_{s,t}\,\mathbb{1}[y_t = \text{next}(s)],$$
where $P_{\text{LM}}$ is the model's ordinary next-token distribution, $\alpha_{s,t}$ is the model's own attention mass on span $s$ summed over its tokens and renormalized over $S_t$, $\text{next}(s)$ is the next token of $s$ given how much of it has already been emitted, and $g_t$ is the LargePiG divergence gate multiplied by $\mathbb{1}[S_t \neq \emptyset]$. Once a span has started, it stays in $S_t$ until it is complete. A copied identifier therefore cannot be cut off partway, which is the fragmentation failure that undid single-token copying in natural-language translation.
The repair loop follows the structure of AutoTransExecuter in CodeTransOcean: translate, execute, feed back the traceback, and regenerate. During regeneration, the source anchors stay active, and the unflagged draft lines become additional copy spans with a fixed prior weight. This gives the repair pass a structural reason to leave correct code alone, in the spirit of the source anchoring that Iterative Translation Refinement with Large Language Models showed is necessary to prevent drift. We build the pipeline on an open-weight decoder-only code model at roughly 7B scale, because the method needs access to attention weights and intermediate hidden states. LargePiG reports about 6% latency overhead at this scale, and we will measure our own overhead.
Three literatures motivate this choice. In summarization, A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization found that reproducing jargon and proper nouns across domains requires pointing. Dr. Summarize penalized vocabulary generation to force verbatim copying of clinical terms. Both show that copying pays off when the copied string must be exact, but both depend on trained pointer layers in summarization, not translation. In technical translation, Global Readiness of Language Technology for Healthcare documents rare terms being replaced by generic substitutes, and SRI's NTCIR-10 PatentMT submission shows subword tokenizers breaking formulas like C_17H_19NO_3 apart. These are the same failure modes in a second domain. Two pieces of prior work constrain the design. The low-resource translation result rules out unrestricted single-token copying. MEIC's "new error trap" (MEIC), where repairs on dense modules introduce new bugs, sets the regression behavior our repair-round anchoring must improve on.
Baselines
The naive reference point is the same open model with plain decoding, in one pass and with the standard compiler-feedback loop run for up to six rounds. Every proposed variant must beat this at equal rounds and equal sampling settings. For calibration only, CodeTransOcean reports Debugging Success Rate (the share of translations that execute correctly within $K$ repair rounds) on LLMTrans for ChatGPT: 48.57% at DSR@0 rising to 52.57% at DSR@3. We will run our open model rather than compare against these numbers directly.
The strongest existing method is kNN-ECD/ECS$_m$ from An interpretable error correction method for enhancing code-to-code translation. It retrieves repair patterns for failing translations from a datastore of execution-verified repairs, after replacing identifiers with placeholders. It reports Java-to-Python accuracy rising from 68.9% for TransCoder-ST to 89.9%. It is the strongest competitor because it also uses tests as an oracle and also targets identifier noise, though it does so by discarding names rather than preserving them. We will rebuild its datastore from the training split of the same benchmark with the same test generator if the original is unavailable, and evaluate on the same held-out test set. Because the two methods intervene at different stages, we will report three conditions: our decoding alone, kNN-ECD applied to plain-decoding outputs, and kNN-ECD applied to our outputs. Beating it means either higher accuracy without a datastore, or a significant gain from stacking. If neither holds, we will report that kNN-ECD dominates.
Two simpler interventions get the same tuning budget as our method, a small held-out sweep. The first is post-hoc identifier renaming: align output identifiers to source identifiers by edit distance and rename. The second is MEIC-style rollback combined with diff-format repairs, where the model emits search-and-replace edits rather than a full rewrite. Three ablations isolate the key ingredient: the LargePiG gate with unrestricted single-token copying, typed anchors without span completion, and span copying with source anchors only (no draft anchors).
Why not just…?
Why not just put the identifier list or glossary in the prompt? In terminology translation the evidence goes the wrong way. GPT-4.1's term accuracy fell from 43.5% to 37.1% when it was given a dictionary (Learning to Translate Ambiguous Terminology by Preference Optimization on Post-Edits), and prompted ChatGPT often ignored glossaries in patent translation (Efficient Terminology Integration for LLM-based Translation in Specialized Domains). Code identifiers may be easier to follow than ambiguous business terms, though, so we run an anchor-list prompt as a baseline rather than assume it fails.
Why not just rename mismatched identifiers after generation? This is the most serious threat to the identifier part of the claim. It is cheap, and it will fix simple renamings. It cannot fix a dropped or merged variable, a mistyped numeric constant, or an altered API name that has no near neighbor among the source identifiers. Our error audit, described below, will report what share of anchor errors are pure renamings. If renaming captures most of the gain, the contribution narrows to literals and to the repair loop, and we will say so.
Why not just emit diffs and roll back on regression? Edit-format repairs together with MEIC's rollback may protect untouched draft lines as well as our draft anchors do. We expect that to happen, and the source-anchor-only ablation is designed to separate the two effects. Diffs do not prevent the first draft from containing an altered identifier, and rollback only restores a state that was itself wrong. We will therefore compare against diff-plus-rollback on both pass rate and per-round regression rate, and combine the two if they turn out to be complementary.
Why not just fine-tune on more parallel code? The patent study found that fine-tuning on a million sentences overfit and made outputs stiff, while 1,000 curated examples did better (Efficient Terminology Integration. Separately, our method is meant to be a training-free addition that can sit on top of any fine-tuned model. We test it on top of a code-specialized fine-tune as well as a base model.
Proposed Experiments
The primary benchmark is LLMTrans (350 execution-verified samples; AutoTransExecuter runs Python targets), measured as DSR@0 through DSR@6. The second code benchmark is the EvoSuite-tested Java-to-Python set from the kNN-ECD paper, measured by test accuracy. Secondary metrics are anchor fidelity, the share of source anchors reproduced exactly where the reference keeps them, and per-round regression rate, the share of tests passing at round $k$ that fail at $k+1$. CodeBLEU is reported only for completeness, because CodeTransOcean found it essentially uncorrelated with execution.
For the transfer test, we run the same mechanism on the WMT Japanese-to-Korean patent task, with chemical formulas, reference numerals, and units as anchors. We report BLEU against the 70.60 achieved by glossary-tuned Mistral-Nemo and the 59.90 of ChatGPT without a glossary, along with exact-match accuracy on anchors. As a control, we run it on a Spanish-to-Catalan pair from the low-resource study, where we predict no gain.
Four checks decide the outcome. The first runs before anything is built: we hand-label baseline failures on LLMTrans by cause. If identifier and literal errors account for under roughly a tenth of failures, the ceiling is too low, and we would refocus on regression control in the repair loop. The second is attribution: the DSR gain should fall mostly on failures labeled as anchor errors. The third is the repair curve: with draft anchors, DSR should keep improving past round three, where CodeTransOcean's curve flattens, and regression rate should fall relative to diff-plus-rollback. The fourth is the inversion test: span-typed copying should help on code and on patent formulas but not on cognate-rich natural language. That pattern would show that the pointer-generator's failure in translation came from copy units that are only near-identical across languages, not from copying as such.
References
- Pointer-Generator Networks for Low-Resource Machine Translation: Don’t Copy That!
- LargePiG: Your Large Language Model is Secretly a Pointer Generator
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- Iterative Translation Refinement with Large Language Models
- A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization
- Dr. Summarize: Global Summarization of Medical Dialogue by Exploiting Local Structures
- Global Readiness of Language Technology for Healthcare: What Would It Take to Combat the Next Pandemic?
- SRI's Submissions to Chinese-English PatentMT NTCIR10 Evaluation
- MEIC: Re-thinking RTL Debug Automation using LLMs
- An interpretable error correction method for enhancing code-to-code translation
- Learning to Translate Ambiguous Terminology by Preference Optimization on Post-Edits
- Efficient Terminology Integration for LLM-based Translation in Specialized Domains