Translation teams keep a database of past human translations, called a translation memory. When a new sentence arrives, they look up the most similar past sentence (a "fuzzy match") and show it to the translation system as a hint. Decades of work report a sharp cliff. Hints above roughly 80% string similarity help a lot. Hints below roughly 70% make output worse than using no hint at all. Large language models (LLMs) prompted with such hints show the same pattern, with very large gains: Mu et al. report BLEU gains of 20 to 30 points over zero-shot on EU legislative text. That text, from DGT-TM and JRC-Acquis, is published openly on the web and almost certainly sits in LLM pretraining data.
This note proposes to test a simple suspicion. Part of the measured value of a high-similarity hint for an LLM may not come from copying the hint at all. The hint may instead work as a cue that triggers recall of a reference translation the model already memorized. If so, the published similarity–utility curve is inflated exactly where it matters most. Any system that learns from that curve how many hints to include per sentence would also learn the wrong policy. We would separate genuine reuse from cued recall by building test sets that are clean by publication date. We would then train a per-sentence hint-count selector on clean and on contaminated data and compare how the two transfer.
Core Claim
The claim is that the fuzzy-match utility curve measured on public benchmarks for LLM translators mixes two mechanisms. The first is reuse: copying and repairing the retrieved target text. This is the "refinement mode" Bulté et al. describe. The second is cued recall: a similar source sentence brings a memorized reference back to the surface.
The two mechanisms predict different curves. On segments the model has never seen, the curve should reflect reuse alone. On memorized segments, the curve should be shifted upward and should show the cliff at a different similarity. The two also leave different fingerprints in the output. Cued recall produces tokens that match the gold reference but appear neither in the hint's target side nor in the model's no-hint output.
The practical consequence concerns dynamic context sizing, meaning choosing per input how much retrieved material to include. A selector trained on contaminated data should over-include mid-similarity hints, which the cliff literature says are harmful on genuinely new text. It should also too rarely choose zero hints.
None of the work in the file tests this. Retrieval-augmented translation papers evaluate on static public splits and assume high similarity means valid guidance: Hoang et al.'s RAT-SI, Cao et al.'s EGNMT, and Mu et al.. Contamination detectors work on question answering, math and code, and look at exact or rephrased overlap, not the 50–80% similarity band: CoDeC and Search-Time Data Contamination. Context-sizing selectors are trained and tested on open-domain question answering, again without asking whether their training labels are inflated by memorization: UtilityQwen, CSM, and FrugalRAG.
Research Program
Contamination-controlled test sets. The first step builds test sets whose contamination status is known rather than guessed, using three independent signals.
The first signal is time. DGT-TM is extended as new EU acts are adopted. Segments from acts published after a model's training cutoff are clean by construction. Their translation memory, drawn from earlier acts, still supplies fuzzy matches across the full similarity range.
The second signal is corpus overlap. For open-weight models whose pretraining corpus is public, such as models trained on Dolma (which CoDeC uses as a testbed), we can check source and reference segments directly for long n-gram overlap with the corpus.
The third signal is behavior. We apply the CoDeC probe to each reference translation. CoDeC reports that unrelated in-context examples lower a model's confidence on memorized targets and raise it on unseen ones, with dataset-level AUC of 99.9%. CoDeC is validated on question answering, math and code, so we treat it as one signal, not ground truth. Its 60–80% "ambiguous zone" is reported as a separate stratum.
Pre-cutoff segments would be labeled seen or unseen only where these signals agree.
Utility curves by similarity and contamination status. On each stratum, we measure translation quality against fuzzy-match similarity, bin by bin, from 0 to 100%. We compare against the no-hint output of the same model. Quality is scored with BLEU and chrF.
Recall fingerprint. We measure the fraction of output tokens that appear in the reference but in neither the hint's target nor the model's zero-hint output.
Decoy hint. This is a causal control. We keep a high-similarity hint's source side and replace its target side with an unrelated sentence of matched length. Under pure reuse, quality should fall to the zero-hint level or below. Under cued recall, a memorized model still produces the reference.
We also address CoDeC's finding head on. CoDeC shows unrelated context disrupts memorized outputs. Our hints are related context, so we predict the opposite sign. We test this by running both unrelated-example and fuzzy-match prompts on the same segments.
Per-sentence hint-count selector. For each sentence and each candidate hint, we compute a leave-one-out label: the change in chrF when that hint is removed from the prompt. This adapts CSM's influence labeling to translation. A small model, initialized as in UtilityQwen with a forward sliding window over the ranked hints, learns to choose $k \in \{0,\dots,K\}$. Here $k$ is the number of hints included in the prompt and $K$ is the largest number of candidates retrieved.
Two copies of the selector are trained under an identical budget, one on labels from contaminated segments and one on labels from clean segments. Both are evaluated on the post-cutoff test set. The prediction is that the contaminated-trained selector picks larger $k$, picks $k=0$ less often, and scores lower on clean data.
Baselines
The naive reference point is zero-shot translation by the same LLM with no hints ($k=0$). Every gain is measured against it. It also defines the cliff: the similarity below which adding a hint is worse than this baseline.
The strongest existing method for LLM translation with hints is TMPLM-style prompting with a fixed number of top hints (Mu et al.). On the law domain they report a BLEU of 64.92, against 61.78 for kNN-MT. We run it with the same retriever, the same candidate pool, the same LLM and the same decoding settings as every other system. Its fixed $k$ is tuned on the same development split our selector uses. Beating it means higher chrF and BLEU on post-cutoff segments at equal or lower average prompt length, with a paired bootstrap test.
For comparison with dedicated translation architectures, we use RAT-SI's fixed $k=5$. RAT-SI reports average gains of 1.43 BLEU (English–German) and 1.73 BLEU (English–French) over plain concatenation. It is evaluated as a prompting setting with five hints, since we are not retraining its encoder.
No reported result was found on Lacuna for any learned per-sentence hint-count selector in translation. Our selector therefore has no published number to beat. Its comparison is entirely against the tuned fixed-$k$ and threshold baselines.
The static similarity cutoff is a single threshold, tuned on development data, that admits all hints above it. It is the reviewer's natural default and is reported separately on clean and contaminated strata.
The ablation that isolates the proposal's key ingredient is the selector trained on contaminated labels. It is identical in architecture, data size and compute to the clean-trained selector and differs only in where its labels come from. If the two perform the same on post-cutoff data, contamination does not matter for context sizing, and the core claim fails.
Why not just…?
Why not just evaluate on post-cutoff data with a tuned static threshold, and skip the selector? This is a real threat. If the clean-data curve has a single sharp cliff at a stable similarity, a well-placed threshold captures most of the benefit. The evidence against it is indirect. Zhang et al. find that the best context size varies with how dense the useful information is, keeping about 11 passages for NQ versus about 5 for HotpotQA. EGNMT extracts useful fragments from matches with only 20–30% similarity, which any cutoff above that would discard. Our experiments settle the question directly: the threshold and the selector are compared on identical post-cutoff data. If the threshold matches the selector, we report that, and the contribution narrows to the measurement result.
Why not just filter by n-gram overlap with the pretraining corpus? Where the corpus is public, we use this, and it serves as one of our three signals. It cannot be applied to closed corpora. CoDeC also reports that overlap checks miss rephrased "shadow" copies, and EU text circulates in many reformatted versions. Agreement between the overlap signal and the time split on open models tells us how much the overlap check misses.
Why not just concatenate every candidate into a long-context prompt? RAT-CAT's deficit relative to RAT-SI shows that hints interfere with each other. Deng et al. find that about 80% of retrieved passages have near-zero influence on question answering. We include full concatenation of all $K$ candidates as an extra row, so its cost and quality on clean data are measured rather than assumed.
Proposed Experiments
The data are DGT-TM and JRC-Acquis for language pairs available in both. We would add at least one lower-resource EU language pair, which addresses the gap left by RAT-SI's evaluation on high-resource pairs only. The models are two open-weight LLMs with public pretraining corpora and one closed model, which receives the time split only.
The first decisive check compares utility curves between strata. A recall-driven inflation would show up as a gap between seen and unseen segments that is concentrated in the 70–95% similarity bins and persists after controlling for sentence length and domain. If the seen and unseen curves coincide, the cliff is a property of reuse alone and the memorization hypothesis is rejected.
The second check is the decoy hint. Quality on memorized segments that stays well above zero-hint quality, while quality on unseen segments collapses, is direct evidence of cued recall. The recall-fingerprint rate should track this effect.
The third check is the selector transfer test on post-cutoff data, with paired bootstrap tests. It compares the clean-trained and contaminated-trained selectors against fixed $k$, the static threshold and full concatenation, on chrF, BLEU, average number of hints included, and the fraction of sentences assigned $k=0$.
The proposal succeeds only if the first two checks show a measurable recall component and the clean-trained selector beats both tuned fixed-$k$ prompting and the static threshold on clean data. A negative result on the first check is still publishable. It would establish that published translation-memory gains for LLMs survive contamination control.
References
- Neural Fuzzy Repair: Integrating Fuzzy Matches into Neural Machine Translation
- Improving Retrieval Augmented Neural Machine Translation by Controlling Source and Fuzzy-Match Interactions
- Learning to Reuse Translations: Guiding Neural Machine Translation with Examples
- Augmenting Large Language Model Translators via Translation Memories
- Detecting Data Contamination in LLMs via In-Context Learning
- Search-Time Data Contamination
- Distilling a Small Utility-Based Passage Selector to Enhance Retrieval-Augmented Generation
- Influence Guided Context Selection for Effective Retrieval-Augmented Generation
- FrugalRAG: Less is More in RL Finetuning for Multi-hop Question Answering