Verified Copying of Retrieved Entity Names in LLM Translation

Machine translation systems get names wrong. Titles of books, films, people and organizations often have culturally fixed translations that cannot be worked out word by word: The Catcher in the Rye becomes Il giovane Holden in Italian. The standard fix is to look the entity up in a multilingual knowledge base such as Wikidata and paste its target-language name into the language model's prompt. This works well when the lookup is correct. When the lookup is automatic and sometimes wrong, which is always the case outside a benchmark, it degrades sharply. The model either ignores the pasted name or adopts it uncritically, including when it is wrong.

We propose to change how the name reaches the output. Instead of hoping the model reads the prompt correctly, we add a copy step to decoding. The model can copy whole target-language names from a small set of retrieved candidates. How much it copies, and which candidate it copies, is weighted by the model's own source-conditioned belief that the candidate is a correct translation. In two sentences: the retrieved names become a menu the decoder copies from, not a hint it reads. The copy is scaled by an internal check of whether the name fits this source sentence, so wrong retrievals get less adoption and correct ones get near-verbatim copying.

Core Claim

The claim is narrower than "pointer-generators help translation." A pointer-generator is a decoder that, at each step, mixes its normal next-word distribution with a distribution that copies tokens from some input, using a learned or computed gate. Our claim is this: in entity-aware translation with automatic retrieval, gated span-level copying from target-language candidates will recover a meaningful share of the gap between gold-ID and end-to-end systems, provided the copy weight is tied to a verification signal. We also claim it will do so without the clean-context penalty that training-based robustness fixes incur.

That gap is large. The SemEval-2025 Task 2 overview (SemEval-2025 Task 2: Entity-Aware Machine Translation) reports a best gold-ID system at M-ETA 89.1 (M-ETA is manual entity translation accuracy, the fraction of entities rendered with a verified target name). The best end-to-end system reaches M-ETA 77.1 with COMET 91.8 (COMET is a learned overall translation-quality score).

We are after one specific failure: bad candidates get adopted. Exposing the Cracks motivates the design. It shows that retrieval-augmented LLM translators adopt retrieved context at high rates, 85% for low-resource pairs versus 65% for high-resource pairs. It also shows that they become more confident rather than less when the context is wrong, and that its adversarial training fix (Adversarial Label Injection) makes models worse at using correct context. That paper concludes models lack a mechanism for checking retrieved material before integrating it. We supply one at decoding time, without training.

Where This Sits Relative to Prior Work

The closest pieces each lack one ingredient.

Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That! added a copy gate over the source sentence in Transformer translation. It found at best marginal gains, and its copy step picked up arbitrary subword fragments. We read this as a constraint on the design, not a refutation. Copying source-language subwords into a target-language sentence is the wrong target. We copy only target-language strings, and only as whole candidate spans, so the fractured-subword failure cannot arise.

LargePiG supplies the core technique. It turns an off-the-shelf LLM into a pointer-generator without training, reading copy weights from self-attention to the context. It sets the copy gate from the divergence between intermediate-layer and final-layer word distributions, because factual tokens keep changing late in the network. It was tested only on monolingual query generation over the prompt's own context, with no retrieval, no multilingual setting, and no notion of a wrong context. We reuse its training-free gate but restrict copying to retrieved candidates and add a verification term, because LargePiG's gate measures "this is a factual token" and not "this retrieved fact is right."

Dialogue systems show the copy-from-external-knowledge pattern works when the source and output share a language. DeepCopy copies hierarchically across multiple persona facts with a switch among sources. Learning to Retrieve Entity-Aware Knowledge and Generate Responses with Copy Mechanism pairs hierarchical retrieval with a knowledge-copy decoder on DSTC 9. Both are monolingual and trained end to end on small models. We borrow DeepCopy's idea of attending across several candidate facts and choosing among them, which doubles as disambiguation.

The translation-side systems all inject knowledge without a structural copy path:

  • Wikidata-Driven Entity-Aware Translation uses prompt injection with gold IDs or with the Relik linker. It reports m-ETA of 0.714–0.728 with Relik, against up to 89.0% with gold IDs, and question-shaped inputs that the model answers instead of translating.
  • CHILL prompts and then self-refines for up to two extra passes.
  • pingan-team bakes entities into a LoRA-tuned Qwen2.5-72B.
  • Knowledge Graphs Enhanced NMT bakes KG entities into weights through pseudo-parallel data.

Decoding-time guidance exists but is unverified or heuristic. Guiding NMT with Retrieved Translation Pieces adds a fixed log-probability bonus for retrieved n-grams. Dynamic Beam Allocation hard-forces constraints. Disambiguated Lexically Constrained NMT shows that over half of constraints are ambiguous and resolves this with a separate trained classifier. Our method differs from all three in the same way: the choice among candidates and the strength of copying come from the translating model's own source-conditioned judgment, computed inside a single decoding pass.

Research Program

The system has three parts. None requires fine-tuning the translator.

Retrieval. An automatic linker, Relik as in Wikidata-Driven Entity-Aware Translation, proposes the top $k$ Wikidata entities per detected mention (we plan $k$ around 5). We collect each entity's target-language label and aliases. All candidates, including wrong ones, go into a delimited candidate block in the prompt.

Verification score. For each candidate string $c$, we compute its length-normalized log-likelihood under the same LLM, given the source sentence and a short "the name X in [target language] is" frame. Crucially, this frame omits the candidate block, so the score reflects the model's parametric belief rather than its tendency to echo context. We combine this with the linker's confidence into a softmax weight $v_c$ over candidates for that mention. This costs one short forced-decoding pass per candidate. That is far cheaper than a full refinement pass.

Copy mixture. At each decoding step $t$ the output distribution is

$$P_t(w) = (1-g_t)\,P^{LM}_t(w) + g_t \sum_{c} v_c\, \alpha_{t,c}\, \mathbb{1}[w = \text{next token of } c],$$

where the symbols are defined as follows:

  • $P^{LM}_t$ is the model's ordinary next-token distribution with the candidate block in context.
  • $\alpha_{t,c}$ is the normalized attention mass the last position places on candidate $c$'s tokens, following LargePiG.
  • "next token of $c$" is defined by tracking, in a prefix tree over candidate strings, how much of $c$ has already been emitted.
  • $g_t$ is a copy gate, set as LargePiG's layer-divergence signal multiplied by $\max_c v_c$.

The multiplication by $\max_c v_c$ is what does the verifying: when no candidate is plausible, copying shuts off and the model translates on its own. The prefix tree forces copies to proceed as whole names, not fragments. A copy can still be abandoned mid-span if $g_t$ falls, which leaves room for target-side inflection in morphologically rich languages. We will report how often that happens.

We will analyze where $g_t$ fires, and whether the copy path reduces the answer-instead-of-translate behavior on question-shaped inputs. The copy path cannot directly force translation, so reduced answering is an observation to measure, not a promise.

Baselines

The naive reference point is the same LLM translating with no retrieved knowledge. On this benchmark Wikidata-Driven Entity-Aware Translation reports m-ETA of roughly 0.257–0.317 for such baselines. This shows how much of the problem is knowledge rather than fluency.

The strongest directly comparable method is prompt-based retrieval augmentation with automatic linking: identical Relik candidates, placed in the prompt, ordinary decoding. It is the natural competitor because it uses exactly our retrieval. The reported figure is m-ETA 0.714–0.728 (Relik pipeline, Wikidata-Driven Entity-Aware Translation). We re-run it ourselves with the same model, the same candidates and the same prompt block, differing only in decoding.

The strongest end-to-end number on the task is SALT's pipeline, at M-ETA 77.1 and COMET 91.8 (SemEval-2025 Task 2). We cannot re-run it, so we treat it as an external target, not a controlled comparison. The gold-ID prompt condition (up to 89.0% m-ETA) and pingan's 89.1 M-ETA / 94.7 COMET gold system serve as ceilings, not competitors.

We also run three decoding-time alternatives on the same candidates:

  • CHILL-style self-refinement, capped at two passes as in CHILL, which reported gains of 0.19–1.66 points in the harmonic mean of M-ETA and COMET.
  • Hard constrained decoding with Dynamic Beam Allocation, forcing the top-linked candidate (Fast Lexically Constrained Decoding).
  • A fixed log-probability bonus on candidate tokens in the style of Guiding NMT with Retrieved Translation Pieces, with its bonus weight tuned on the same development split and with the same number of trials as our method's one free hyperparameter, the verification temperature.

Two further baselines isolate the key ingredients:

  • Rerank-then-prompt. Use our verification score to keep only the top candidate, then prompt normally. This isolates the value of the copy path from the value of verification.
  • Unverified copy. Run the copy mixture with $v_c$ uniform and the gate set by layer divergence alone, which is essentially LargePiG pointed at the candidate block. This isolates the value of verification from the value of copying.

Beating the controlled prompt baseline means a higher end-to-end M-ETA at equal or better COMET, averaged over the ten directions, with paired bootstrap significance. We would count a gain of at least three M-ETA points as practically meaningful, roughly a quarter of the gold/end-to-end gap.

Why not just…?

Why not just put the retrieved names in the prompt and trust a strong LLM? With gold IDs this nearly saturates the benchmark (up to 89.0% m-ETA). So on clean retrieval our method has little to add, and we expect only to match it. The problem is automatic retrieval. There the same prompting drops to 0.714–0.728 (Wikidata-Driven Entity-Aware Translation), and Exposing the Cracks shows models adopt wrong context at high rates while growing more confident. The controlled comparison at identical retrieval settles whether decoding-side verification helps.

Why not just rerank the candidates better and then prompt? This is the real threat. If the verification score alone picks the right candidate, a prompt with only that candidate might capture the whole gain, and the copy mixture would be unnecessary machinery. The rerank-then-prompt ablation tests exactly this. We expect copying to matter in two places: when the model is given the right name but still paraphrases it, which is the "you can't just throw entities and hope" failure reported by CHILL, and when soft weighting over two close candidates beats a hard top-1 choice. If rerank-then-prompt matches the full system within noise, we will report that the contribution is the verification score, and drop the pointer claim.

Why not just force the name with constrained decoding? Hard forcing guarantees the string appears, but it assumes the constraint is correct. Over 50% of constraints are ambiguous according to Disambiguated Lexically Constrained NMT, and forced constraints push the decoder off its preferred path (Fast Lexically Constrained Decoding). With noisy linking, forcing a wrong top-1 name is the worst case. The injected-noise experiment below measures this directly.

Why not just fine-tune with LoRA on Wikidata-augmented data, as the winning team did? pingan-team shows this is very strong, with a leaderboard score of 91.93. But the knowledge becomes fixed in the weights and was evaluated with gold IDs. Our question is test-time use of retrieval that may be wrong, and of entities absent from training. We will include a held-out split of entities not seen in any fine-tuning data, where parametric storage cannot help. Our method is orthogonal and could run on top of a tuned model.

Decisive Experiments

All experiments use XC-Translate from SemEval-2025 Task 2: ten directions from English, roughly 5,000 test sentences per pair. We use open instruction-tuned Qwen2.5 models at 7B, the scale LargePiG was tested at, and at 72B if compute allows, since LargePiG left scaling open. We report M-ETA, COMET, their harmonic mean, and wall-clock latency.

End-to-end comparison. The first check is the main end-to-end comparison against the controlled Relik prompt baseline and the ablations above. It answers whether verified copying closes part of the gap, and whether the gain comes from verification or from copying.

Injected wrong candidates. The second check borrows the noise design of Exposing the Cracks and applies it to entity candidates. Starting from gold IDs, we insert or substitute wrong candidates of three kinds: a literal word-by-word translation of the title, a different entity with the same surface name, and an unrelated entity. We then measure M-ETA and an adoption rate, the fraction of outputs containing the injected wrong name. The paper found adoption rates of 85% versus 65% between low- and high-resource pairs. That finding predicts our largest adoption reduction on the lower-resource directions in this benchmark, such as Thai and Turkish, relative to French and German. Seeing that pattern would support the mechanism; a flat pattern would weaken it.

Clean-context guard. The third check guards against the trade-off that Adversarial Label Injection showed. On gold candidates, the method must stay within noise of the gold prompt condition. A robustness gain bought with a clean-context loss would count as failure.

Mechanism and cost checks. We report latency against CHILL-style refinement and against the prompt baseline; LargePiG's copy step alone added about 6% latency on a 7B model. We also compare the rate of answer-instead-of-translate outputs on question-shaped source sentences across conditions, as a measured side effect rather than a claimed fix.