Whole-Word Speculative Copying for Closely Related Low-Resource Languages

When a translation system goes from Hindi into a close relative such as Bhojpuri, Magahi, Marathi or Sanskrit, many output words are cognates, names or loanwords. They can be copied from the source sentence or looked up in a small memory of past translations instead of being generated letter by letter. Copy mechanisms were built for exactly this situation. In practice, though, they copy meaningless subword fragments. Retrieval from external memory also helps, but it slows decoding by a factor of three or more. We propose to handle both problems with one decoding procedure. Candidate words are proposed whole, as word-sized chunks found by a learned boundary detector, from both the source sentence and a retrieved memory. The translation model then checks the whole candidate in a single forward pass and either accepts all of it or rejects all of it.

Each accepted chunk is a copy decision made at word granularity, so the model can never paste half a word. It also emits several tokens for the price of one forward pass, which pays for the retrieval. Under this scheme copying improves quality, retrieval stops costing time, and segmentation determines what a "word" means in scripts where whitespace misleads.

Core Claim

The weak results for pointer-generators in low-resource translation come from copying at the wrong granularity, not from copying itself. A pointer-generator is a decoder that, at each step, mixes "generate a word from the vocabulary" with "copy a word from the input" through a learned switch. Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That! added such a switch to Transformer translation for pairs including Hindi–Marathi and Hindi–Bhojpuri. It found at best marginal gains, slightly worse results on sentences with heavy subword overlap, and copying of random fragments. The authors blamed byte-pair encoding (BPE), the standard frequency-based subword tokenizer, but did not test the fix. Byte-pair encoding over-fragments these languages badly: When Tokenizers Fail reports that the Qwen3 tokenizer splits 97.7% of Bhojpuri words, at 4.43 tokens per word.

We claim that if copying is restricted to whole word-like chunks and decided by verification against the model's own distribution, then copying helps on exactly the high-overlap sentences where it previously hurt. The same mechanism should also cut wall-clock decoding time below that of the plain model, rather than adding the 3× retrieval tax measured for kNN-TL.

How This Differs From Closest Work

The two retrieval-as-acceleration papers supply our basic technique and constrain how we use it. REST proposes continuations from a suffix-array datastore and verifies them in parallel, reaching up to 2.36× speedup with lookups under 1 ms per token. NEST retrieves variable-length spans and uses relaxed verification, which yields both speed (1.8× on LLaMA-2-Chat-70B) and factuality gains. Both work on English, at subword granularity, and accept the longest matching token prefix. That rule is the fragment-copying failure in a new form, because a draft can be cut off mid-word.

We change three things. Drafts are bounded by learned word boundaries. Acceptance is all-or-nothing per chunk. The relaxation threshold is treated explicitly as a copy switch that can be calibrated against the target model's distribution.

The boundary detector is taken from When Tokenizers Fail. That work's byte-level H-Net front end predicts word-aligned chunks from similarity between neighbouring byte representations, and it was evaluated only on tagging and classification. We use the detector only to mark draft boundaries. We do not feed chunk embeddings into the translation model. This sidesteps that paper's fragile mapping into a frozen subword vocabulary and its finding that chunking does not help document-level semantics.

The retrieval memory follows Multilingual k-Nearest-Neighbor Machine Translation, where low-resource languages borrow from related high-resource languages (+3.6 BLEU on Bosnian–English, 5.3× faster with family-specific stores). It also follows kNN-TL, which uses a parent-language datastore. Both mix memory into every token's probabilities, paying a search at every step. We query memory once per source word and use it only to propose drafts.

Training-time injection in INK removes the datastore but leaves a documented "hybrid gap": keeping memory at inference still helps. Our design keeps memory at inference and aims to keep it from costing time.

Research Program

The translation model is NLLB-200 distilled 600M (No Language Left Behind), fine-tuned on Hindi→X bitext. The bitext comes from NLLB mined data and CVIT-PIB, the same sources used by the Don't Copy That study.

A small byte-level chunker is trained with the recipe from When Tokenizers Fail: Hindi FineWeb2 data plus its chunk-alignment loss, which that paper shows is necessary because without it chunks span 8–17 words. The chunker segments the source sentence, and also segments the target side of the training bitext.

From the segmented bitext we build a word-chunk memory. Keys are source chunks plus a little left context. Values are the aligned target chunks found by an off-the-shelf word aligner. Lookup is exact or edit-distance-bounded string match, not dense vector search. This keeps the cost in REST's sub-millisecond regime, and it is robust to small spelling differences between cognates.

At each decoding step the candidate pool has three parts: source chunks near the current attention focus (identity copies), memory values for those source chunks, and memory continuations of the current target suffix. Candidates are arranged in a prefix tree and verified in one forward pass with tree attention, as in REST.

A candidate chunk $c$ spans subword positions $1..m$ under the model's tokenizer. It is accepted only if every position passes

$$p_\theta(c_j \mid y_{<t}, c_{<j}, x) \;\ge\; \lambda(h_t)\,\max_{v} p_\theta(v \mid y_{<t}, c_{<j}, x), \qquad j = 1..m.$$

Here $x$ is the source, $y_{<t}$ is the output so far, $p_\theta$ is the model's next-token distribution, and $\lambda(h_t)\in(0,1]$ is a copy threshold computed from the decoder state $h_t$. If any position fails, the whole chunk is rejected and the model generates one token as usual.

With $\lambda=1$ the output matches greedy decoding exactly and only speed changes. Lowering $\lambda$ biases the system toward copying, which plays the role of the copy switch. Penalizing generation to force copying was done by hand in Dr. Summarize. We parameterize $\lambda$ with a tiny gate, in the spirit of the under-1,600-parameter copy layer of A Copy-Augmented Generative Model for Open-Domain Question Answering. The gate is trained on held-out bitext to maximize chrF++ of accepted chunks under a length penalty, with the base model frozen.

The analysis has three parts. The first measures copy precision: the fraction of accepted chunks that appear in the reference, compared with fragment copying under a standard pointer-generator. The second stratifies results by the source–reference shared-word density bins used in the Don't Copy That study. The third relates acceptance rate and mean accepted length to wall-clock speed.

Baselines

The naive reference point is the fine-tuned NLLB-200 600M with greedy and beam-5 decoding and no copying or retrieval. Every other system starts from this checkpoint and uses the same bitext. This anchors both quality and latency.

The most direct competitor on the copying question is a standard pointer-generator head added to the same fine-tuned model over byte-pair-encoding tokens, reproducing Don't Copy That!. It is trained on identical data with the same hyperparameter search budget as our gate (a fixed number of trials on the same dev set). That paper reports no numerical BLEU values in the file, so no reported result was found on Lacuna for pointer-generator BLEU on Hindi–Bhojpuri. Beating it means higher chrF++ overall, and in particular on the high-overlap bin where it underperformed the plain model.

The strongest retrieval competitor is token-level nearest-neighbour interpolation with a pooled Hindi-family datastore, following Multilingual k-Nearest-Neighbor Machine Translation and kNN-TL. Its datastore is built from exactly the bitext behind our memory. The closest reported figures are +3.6 BLEU on TED Bosnian–English (Multilingual kNN-MT) and 20.6 BLEU on Indonesian–English versus 19.7 for ConsistTL, alongside roughly 3× slowdown (kNN-TL). No reported result was found on Lacuna for nearest-neighbour MT on Hindi→Bhojpuri, Magahi or Sanskrit. Beating it means matching its chrF++ within confidence intervals while running faster than the naive baseline, which it cannot do.

The strongest acceleration competitor is REST-style drafting over the same source and memory text at subword granularity with longest-prefix acceptance (REST), at both $\lambda=1$ and the same relaxed $\lambda$ as ours. This isolates the key ingredient, whole-word chunks. Further ablations replace learned chunks with plain whitespace words, remove the memory (source-only drafts), and fix $\lambda$ instead of learning it.

For context, we also run Chain-of-Dictionary prompting (Chain-of-Dictionary Prompting) on an open instruction-tuned LLM with the same lexicon. That paper reports English→Serbian Cyrillic rising from 3.08 to 42.63 chrF++ with ChatGPT, with up to 1.8× inference time.

Why not just…?

Why not just replace the tokenizer and keep a standard pointer-generator? Efficient Low-Resource Language Models Using Tokenizer Transfer shows that a pretrained model can be moved to a new tokenizer, and the Don't Copy That authors suggest exactly this. This is a real threat to the quality half of our claim. We therefore run it: NLLB transferred to a morpheme-aware tokenizer trained on Hindi/Bhojpuri/Marathi, plus a pointer head. If it matches our chrF++ and copy precision, the quality gain comes from granularity alone. Our remaining contribution would then be speed, plus avoiding retraining embeddings on very little data.

Why not just fine-tune NLLB longer or with more back-translated data? More data helps whatever decoder sits on top. Our method is a decoding procedure applied to that same checkpoint, so stronger fine-tuning raises both arms. The test is whether the gap over the naive baseline persists when both start from the best fine-tuned checkpoint. Fine-tuning alone also gives no speedup.

Why not just use REST or prompt-lookup drafting as is? At $\lambda=1$ subword drafting is lossless and fast, so it may match our speed. It cannot improve quality, though, and with relaxed acceptance it reintroduces mid-word copying. The subword-REST arm with matched $\lambda$ settles whether whole-word acceptance matters. We expect the difference to be small for Bhojpuri, where whitespace roughly tracks words, and large for Sanskrit and Marathi, where sandhi and agglutination break that link. If no difference appears even there, the boundary component should be dropped.

Why not just prompt an LLM with a dictionary? Chain-of-Dictionary gains are large but come with tripled prompts and up to 1.8× latency. The model can also ignore the dictionary, since nothing forces it to use the entries. We report chrF++ and latency for both so that the trade-off is visible rather than assumed.

Proposed Experiments

Evaluation uses FLORES-200 devtest for Hindi into Bhojpuri, Magahi, Maithili, Awadhi, Marathi and Sanskrit, with chrF++ as the primary metric and spBLEU as secondary. Off-target language rate is measured with a language identifier. Latency is wall-clock time per sentence at batch size 1 on one GPU, together with mean accepted chunk length and acceptance rate.

The decisive checks are threefold. On the high shared-word bin, our method at its learned $\lambda$ must beat the naive model, reversing the pointer-generator result. At $\lambda=1$ it must be lossless and faster than greedy decoding, while the nearest-neighbour baseline is slower. The learned-chunk versus whitespace versus subword ablation must show a granularity effect that grows with how badly whitespace mismatches morphology, from Bhojpuri to Sanskrit.

A further check addresses the risk that relaxed acceptance merely inflates copying. Copy precision on accepted chunks must exceed the model's own token accuracy at those positions, or else lowering $\lambda$ is just trading fluency for overlap with the source.