Teaching a Summarizer to Copy Entities It Has Never Seen

When a multilingual text generator is fine-tuned on English and then asked to summarize a Hindi or Tamil article with no Hindi or Tamil training data, the failures are rarely about fluency. They are about names. The model drops a person's name, swaps in a more famous one, writes it in English, or produces something phonetically close but wrong. The standard fix for rare names in generation is the pointer-generator. This is a decoder that, at each step, can either produce a word from its vocabulary or copy a word directly from the input. The fix has a bad record across languages. A careful study of low-resource translation found that copying helped only marginally and that it copied seemingly random word fragments (Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That!).

This note proposes two changes. First, copying should operate on whole entity spans rather than subword fragments. Second, the model should learn when to copy through entity masking during English training: each name in the source and target is replaced with a consistent random string, or with an alias in a foreign script. A name the model cannot have memorized can only be reproduced by copying. The bet is that this copy policy is learned from context rather than from English word identity, so it should transfer to unseen languages and scripts. If it does, zero-shot summaries in low-resource languages should contain far fewer hallucinated or mistranslated entities, at almost no parameter cost.

Core Claim

The negative result for cross-lingual copying is a claim about two things: what gets copied (arbitrary subwords) and how the copy decision is learned (from incidental string overlap). It does not show that copying itself is useless across languages. We claim that span-level entity copying, trained under entity masking, gives a copy gate that fires at entity positions in languages the decoder never saw during fine-tuning. We test this in zero-shot cross-lingual summarization, where it should reduce entity hallucination substantially relative to the same model with no copy mechanism or with ordinary token-level copying.

Summarization is the main test bed, not entity linking or extractive question answering. In those tasks the output is either already constrained to a list of known titles or is literally a span of the input, so a copy mechanism adds little. Abstractive summaries are different: they must weave copied names into freely generated text. That is the situation where generation and copying actually compete.

Mechanism

Let $E$ be the set of entity spans detected in the source document. The detector is a multilingual tagger trained on WikiAnn with entity-centric code-switching, following EntityCS. At decoding step $t$, the decoder can either emit a vocabulary token or emit an entire source span $s \in E$ in one action:

$$P(y_t) = p_{gen}\,P_{vocab}(y_t) + (1-p_{gen})\,\alpha_s \,[y_t = s],$$

The terms are as follows. $p_{gen} \in [0,1]$ is the learned gate computed from the decoder state. $P_{vocab}$ is the usual distribution over vocabulary tokens. $\alpha_s$ is the decoder's cross-attention summed over the tokens of span $s$ and renormalized over all spans in $E$. The bracket $[y_t = s]$ restricts the copy term to the event that the output is span $s$.

This differs from the token-level mixture analyzed in the low-resource translation study in two ways. Probability mass for copying can only land on entity spans, and a copy action reproduces a complete name, so the model cannot copy fragments. Following A Copy-Augmented Generative Model for Open-Domain Question Answering, which built a pointer into Fusion-in-Decoder with only 1,537 extra parameters by reusing cross-attention, we reuse existing attention and add only the gate.

Surface form variation enters through morphology. In Hindi, Turkish or Tamil, a summary may need a name in a different case form than the one in the source. We therefore allow a copied span to be followed by generated suffix tokens. We report how often the copied form plus generated suffix matches the reference's inflected form, and how often the model needed a suffix the source lacked.

Entity Masking as the Training Signal

The gate needs examples where copying is the only correct move. On ordinary English data such examples are rare, because the vocabulary distribution can usually produce "Obama" unaided. We create them in two ways.

The first is zero-coverage mention randomization, after Hongyu Lin et al.'s work on named entity recognition as described in the file. On a fraction of English training pairs, every entity is replaced with a unique random string, applied consistently in source and reference.

The second is cross-lingual alias substitution. Each entity is replaced with its Wikidata label in a randomly chosen other language and script, using the alias mappings that EntityCS used to build its 231-million-sentence code-switched corpus.

EntityCS used such substitutions to improve an encoder through masked prediction. It reports a +2.8% gain on WikiAnn and a +10% absolute gain on X-FACTR. We use the same substitutions for a different purpose: to make a decoder's copy decision indifferent to the script and identity of the name.

The second variant matters because of What Drives Performance in Multilingual Language Models?. That analysis found that for languages unseen in pretraining, script type was the dominant performance driver in 23 of 31 model settings. A copy gate trained only on Latin-script random strings may not fire on Devanagari or Ge'ez names.

Research Program

We fine-tune on English summarization only and evaluate zero-shot on other languages. We use two base models so the result is not tied to one tokenizer. The first is IndicBART, which maps 11 Indic scripts to Devanagari and therefore makes entity spans more shareable across related languages. The second is mBART-50, which keeps native scripts. Comparing them separates the copy mechanism's contribution from that of script unification.

The primary benchmark is XL-Sum, using its English portion for training and its Indic and other non-English portions for zero-shot evaluation. The secondary benchmark is WikiLingua, which offers aligned how-to articles across 18 languages and lets us test languages outside the Indic family.

We report ROUGE, but ROUGE is not the deciding metric. The headline metrics are entity-focused:

  • Entity hallucination rate: the fraction of entities in the output, as detected by the same tagger, with no fuzzy or transliteration-normalized match in the source.
  • Entity recall: the fraction of reference entities reproduced.
  • Entity language fidelity: the fraction of output entities written in the source script rather than English or another language.

Because tagger errors on unseen languages could flatter or hide the mechanism, every evaluation is repeated with oracle entity spans on a manually annotated subset of a few hundred documents per language.

Baselines

The naive reference point is the same pretrained model fine-tuned on English summaries with no copy mechanism and no masking, applied zero-shot. It shows how much entity hallucination comes for free with implicit copying through attention.

The second baseline is a standard token-level pointer-generator on the same model, trained on the same data. This is the design that Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That! found gives only marginal gains. It is the most direct competitor because it is what anyone would try first. Our method must beat it on hallucination rate across scripts, not only within the Latin script.

The third baseline is a translate-test pipeline: translate the source into English, summarize, and translate back. It represents the pipeline approach that WikiLingua found weaker than direct models on Turkish-to-English, with 41.06 ROUGE-1 against 42.76 for its best direct system. Those figures come from a different cross-lingual setting and are cited only as evidence that pipelines are beatable. We found no reported result on Lacuna for zero-shot English-to-Indic XL-Sum summarization, so all zero-shot numbers will be produced by us under identical conditions.

The ablations isolate the two ingredients:

  • span-level copy without masking;
  • masking without any copy mechanism;
  • masking with token-level copy;
  • random-string masking only versus cross-lingual alias masking.

All systems share the same base checkpoint, English training data, number of optimizer steps, and hyperparameter search budget of eight configurations each, selected on English validation only. Beating the competitors means a statistically reliable reduction in entity hallucination rate, by bootstrap over documents, relative to the best competing system on a majority of evaluation languages, including at least two non-Latin scripts not seen in fine-tuning, without a loss of more than one ROUGE-L point.

Why not just…?

Why not just use a standard pointer-generator and let it copy subwords? That experiment has been run, and it disappointed. Copying gave marginal gains, no extra benefit for closely related language pairs, and slightly worse results on subsets with high subword overlap. The copy decision also did not track attention entropy (Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That!). We keep this system as a baseline. We also measure whether our gate's copy decisions line up with entity boundaries and attention, which is exactly the relationship that study found missing.

Why not just apply entity masking and rely on the pretrained decoder's implicit copying? This is the most serious threat. Pretrained sequence-to-sequence models already copy well through cross-attention. Autoregressive Entity Retrieval reports that GENRE retrieved 100% of cold-start entities whose name exactly matched the mention. Randomized names alone might teach that implicit behavior without any gate. The masking-without-copy ablation settles this directly. If it matches the full system, the honest conclusion is that masking is the active ingredient and the span copy action is unnecessary, and we would report it that way.

Why not just constrain decoding so entity tokens must come from source spans, with no learned gate? A hard prefix-tree constraint built from the source's entity spans, in the style of Multilingual Autoregressive Entity Linking, would eliminate fabricated names by construction. But the decoder would still need to know when it is inside an entity. Hard constraints also cannot accommodate the generated inflectional suffixes discussed above. We include this as an additional baseline. It counts as a real alternative if it matches our hallucination rate without harming inflection match and ROUGE.

Why not just use a larger multilingual model? Scale may reduce hallucination, but the SIB-200 analysis found that model size did not change the fact that script dominates unseen-language performance (What Drives Performance in Multilingual Language Models?). IndicBART matched or exceeded the much larger mBART-50 by up to 2 BLEU in machine translation. We test whether the gap between our method and the naive baseline shrinks when moving from the IndicBART-sized model to mBART-50. A shrinking gap would suggest scale partly substitutes for the mechanism.

Decisive Checks

Three outcomes would make or break the claim.

  1. Script transfer of the gate. The gate's copy rate on entity positions should stay high on scripts absent from fine-tuning. It should be markedly higher under alias masking than under random-string masking. This would show that the copy policy transfers across scripts, rather than merely being learned in English.
  2. Persistence with oracle spans. The hallucination reduction must survive the switch to oracle entity spans, ruling out an artifact of tagger errors.
  3. Generative question answering as a stress test. On TyDi QA, where mBERT's extractive reader reaches 50.5 F1 on minimal answer spans, a zero-shot generative reader with span copy should close most of its gap to an extractive XLM-R reader trained identically. The same comparison is run on MLQA, whose reported XLM Hindi score is 48.8 F1. Extractive readers copy by construction, so they set the ceiling for answer fidelity here.

If span copy reaches that ceiling while remaining a generator, the mechanism does what it promises. If it falls short even with oracle spans, the "Don't Copy That" conclusion extends to entities as well.