When a model summarizes an English article in Chinese, or answers a Bengali question from an English passage, it must carry names, numbers and technical terms across languages. It must also keep straight who did what, and whether something happened or did not. Monolingual summarizers solve the first problem with a copy mechanism: at each output step, a learned switch decides whether to generate a word from the vocabulary or to copy a word directly from the input. Across scripts, copying the input string is useless, because an English name is not a Chinese token. Copying also creates a second, quieter problem. It rewards outputs that reuse the source's words, and models that reuse words tend to treat "same words" as "same meaning." An output can contain every correct entity and still reverse the relation between them.
We propose a copy mechanism that copies from a target-language shadow of the source. Before decoding, each salient source span (an entity, a number, a term, or a short negated predicate) is aligned and rendered into the target language, and the decoder copies from these renderings while attending to the original spans. We also train the copy switch on paired, minimally perturbed inputs: negations, argument swaps, quantifier flips. A model that copies the right entities into the wrong roles is then penalized rather than rewarded. The central claim has two parts. First, cross-lingual copying fails because of what gets copied (arbitrary subword strings), not because copying is a bad idea. Second, any copy mechanism will import the lexical-overlap shortcut into generation unless the training forces it to separate the two.
What is known, and where this sits
The copy mechanism (a "pointer-generator") mixes a vocabulary distribution with the attention distribution over input positions, weighted by a learned gate $p_{gen}$. The survey Pretrained Language Model for Text Generation: A Survey describes this formulation. Monolingual work motivates using it for terminology. A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization finds that domain-specific terms need explicit pointing. Dr. Summarize penalizes the vocabulary distribution to force extraction of clinical entities. A Copy-Augmented Generative Model for Open-Domain Question Answering grounds answers in retrieved passages with a copy layer of under 1,600 parameters. All three copy within a single language.
The cross-lingual case has a direct negative result that constrains our design. Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That! adds a standard copy mechanism to translation between closely related languages such as Spanish–Catalan and Hindi–Marathi. It finds at best marginal BLEU gains and slightly worse results on test slices with high shared-subword density. Its copy decisions track arbitrary subword fragments rather than cognates. We take this as evidence against copying source subwords, not against copying in general, and that paper did not test any alternative copy target. Two papers show that a pointer can target something other than the source string, and we borrow that technique. Concept Pointer Network for Abstractive Summarization adds a second pointer into an English concept taxonomy to counter "lazy copying," raising novel unigrams on Gigaword from 14.3% to 17.2%. AMR Parsing with Action-Pointer Transformer points into an action history. Both are monolingual. Our pointer targets per-span target-language renderings instead of a fixed English taxonomy.
The alignment step relies on Cross-lingual AMR Aligner: Paying Attention to Cross-Attention. It shows that the cross-attention of an mBART parser concentrates alignment information in an intermediate layer, around layer 3, and yields word-to-node alignments roughly 40 F1 better than earlier aligners for German, Spanish and Italian. LargePiG shows that a frozen decoder-only LLM can act as a copy mechanism. It reads source attention for the copy distribution and sets the gate from the divergence between intermediate-layer and final-layer predictions. It is evaluated only within English and within Chinese, so it serves as our training-free baseline rather than our method.
The lexical-overlap literature supplies both the motivation and the training recipe. PAWS shows BERT falling from over 90% on Quora pairs to 33–40% on word-scrambled pairs, and recovering to 85% after adding about 12,000 scrambled examples. PAWS-X carries this to six languages; mBERT averages 87.2% but lags around 83–84% on Chinese, Japanese and Korean. SemAntoNeg shows a popular sentence encoder scoring 1.9% against a 33.3% random baseline when negation and antonymy are at stake, while NLI-trained models reach 93.6%. Schlegel and Nenadic report that quantifier and modal flips remain hard even after such training. These are classification results. Nobody has asked whether a generator with a copy gate inherits the same shortcut. A copy gate is close to a learned "overlap detector," and ROUGE-style training and evaluation reward overlap directly, so we expect it does.
One result could be read as cutting against us. Cross-Lingual Ability of Multilingual BERT shows that removing all subword overlap costs only 1.4 XNLI points, so encoders do not need lexical sharing to transfer. We agree, and that is part of the argument. Cross-lingual understanding does not run on surface overlap. Cross-lingual generation still has to emit exact strings for names and numbers, and the factuality audit in Evaluating Factuality in Cross-lingual Summarization finds over half of outputs factually inconsistent, with terminology, particulars, tense and plural errors prominent. The copy mechanism should carry exact strings and nothing else. The relational content should come from the shared structural representation that the mBERT study identifies.
Core Claim
A cross-lingual copy mechanism helps when it copies target-language renderings of aligned source spans. It reduces terminology and particulars errors relative to end-to-end, pipeline and naive-copy systems. We also predict that, trained the usual way, it will raise polarity and role errors: it will copy "guilty" without "not," or "London" and "New York" into swapped slots. Training the gate on minimally perturbed source pairs, each paired with a correspondingly changed target, removes most of that increase without giving back the entity gains. The second half is the more surprising claim and the one we most want to test.
The Mechanism
Given a source document, we extract candidate spans: named entities, numerals, dates, rare terms, and short predicate spans that include their negation or quantifier. Including the operator is deliberate, so that "not guilty" is copied as a unit. Each span $s_j$ gets a target-language rendering $r_j$ from a fixed renderer. The renderer is the backbone model run in translation mode, constrained by a bilingual lexicon or transliterator where one exists. The renderings are concatenated into a shadow memory of target-language tokens, each token tied to its source span. The decoder's output distribution is
$$P(y_t = w) = p_{gen}\,P_{vocab}(w) + (1 - p_{gen}) \sum_{k:\, m_k = w} b_{t,k},$$
where $m_k$ is the $k$-th token of the shadow memory and $b_{t,k}$ is the attention weight that decoder step $t$ places on it. This is the standard copy mixture, except that the copied string is already in the target tokenizer, so the subword mismatch identified in Don't Copy That! cannot arise. The attention key for each shadow token combines its own embedding with the encoder representation of its aligned source span. The pointer therefore chooses where to copy from based on source context, while what it emits is target-language text. We supervise this attention with the guided cross-attention loss from the AMR aligner work at the layer that paper identifies.
For training, a fraction of each batch is replaced by perturbed pairs. We inject a negation, swap two arguments of a relation, or flip a quantifier in the source, then edit the reference target to match, using rule-based edits checked by a cross-lingual NLI model and spot-checked by annotators. The perturbed and original pairs share almost all spans, so a gate that copies by surface identity gets the same copy reward for both. Only a model whose generated relational frame follows the source can fit both. This is the generation analogue of the PAWS augmentation.
Research Program
The work has three parts. First, we build the shadow-memory copy decoder on a multilingual encoder-decoder backbone (mBART-50 or mT5-base), and a decoder-only variant on a multilingual LLM that receives the shadow memory in context. Second, we build a cross-lingual perturbation probe for generation. It consists of paired source documents differing by one semantics-altering edit, drawn from the En2ZhSum and Zh2EnSum test sets of NCLS and from XOR-TyDi passages, with types modeled on SemAntoNeg (negation and antonymy) and PAWS (role swap), plus the quantifier and modal flips that Schlegel and Nenadic flag. Third, we use this probe to measure whether copying trades entity errors for relational ones, and whether perturbation-paired training removes the trade.
Baselines
The naive reference point is an end-to-end multilingual seq2seq model fine-tuned for cross-lingual summarization with the NCLS multitask recipe (summarization plus translation), on the same backbone and data with no copy mechanism. NCLS reports that this setup beats the best pipeline by 4.87 ROUGE-2 on En2Zh and gains another 3.60 ROUGE-2 from translation multitasking. The file gives only these relative gains, and no absolute ROUGE-2 value for NCLS appears on Lacuna. We therefore re-train it ourselves rather than quote a target.
The cascaded baseline is translate-then-summarize: the same backbone in translation mode feeding a monolingual copy-augmented summarizer, with matched total training compute.
The naive-copy baseline is a standard pointer-generator over shared multilingual subwords, as in Don't Copy That!. That paper reports only qualitative "marginal" gains, so we re-run it on our data.
The strongest training-free baseline is LargePiG applied cross-lingually to a multilingual LLM. It reports beating DoLa on TruthfulDQG and TruthfulVQG with about 6% added latency, but no numeric truthfulness scores are on Lacuna and it was never run cross-lingually.
The strongest competitor overall is a prompted multilingual LLM of comparable or larger size that receives the same shadow memory as an in-context glossary. This tests whether the learned gate adds anything beyond the renderings.
For open-retrieval QA, the reported reference is the best XOR-Full system at 18.7 F1 from XOR QA. There we compare under the same retriever.
Two ablations isolate the key ingredients. The first removes the pointer but keeps the shadow memory as extra encoder input, or enforces it through lexically constrained decoding; it isolates the learned copy gate. The second keeps the pointer but removes perturbation-paired training; it isolates the shortcut fix.
All systems share training data, backbone size where applicable, and a fixed hyperparameter search of equal trials. Beating the field means two things. The full model must lower human-annotated terminology and particulars error rates on a Fact_CLS-style audit relative to the best non-copy system. It must also not raise, and ideally lower, the rate of polarity and role failures on the perturbation probe relative to the no-copy baseline. Gains in ROUGE alone do not count.
Why not just…?
Why not just put the glossary in the prompt of a large multilingual LLM? This is the most serious threat. If the renderings carry most of the benefit, a prompted LLM may match a trained copy gate on entity accuracy. Scale has not removed overlap blindness in encoders: SemAntoNeg finds general-purpose models failing on negation regardless of downloads, with only NLI-trained models succeeding. That evidence concerns classifiers, though, not generators. The glossary-in-prompt baseline is therefore run head-to-head. If it equals the full model on both entity errors and the perturbation probe, the contribution reduces to the shadow memory plus perturbation training, and we would report it that way.
Why not just use lexically constrained decoding with the same span translations? Constrained decoding guarantees that the right strings appear but says nothing about where they go. We expect it to score well on entity accuracy and poorly on role swaps, which is precisely the overlap shortcut made explicit. It is our first ablation, and the probe decides the question.
Why not translate first and then summarize monolingually? NCLS reports pipelines losing 4.87 ROUGE-2 to end-to-end training. The Fact_CLS audit finds both families inconsistent more than half the time, and XOR-Full's best cascaded system reaches 18.7 F1. We still run the pipeline with matched compute, because a strong modern translator could narrow the gap.
Why not just add a standard copy mechanism to a multilingual model? Don't Copy That! already shows this yields marginal or negative gains because copy decisions track arbitrary subwords. We include it to confirm that our gains come from changing the copy target rather than from adding a copy gate.
Proposed Experiments
We train on En2ZhSum and Zh2EnSum and evaluate in four ways. ROUGE is a sanity check. A human audit of 200 documents per direction uses the Fact_CLS ten-type error taxonomy, which is why we audit fresh outputs rather than reuse its annotations. Entity, number and date exact-match against references is checked separately for cases where the renderer was wrong, so we can see whether the model blindly propagates rendering errors. The perturbation probe reports flip sensitivity: the fraction of paired inputs where the output's polarity or argument roles change in agreement with the source edit. This is judged by bilingual annotators on a subset and by a cross-lingual NLI checker on the full set, and we report the agreement between the two. We repeat the evaluation on XOR-TyDi XOR-Full (F1 against the 18.7 reference) and on MLQA's generalized cross-lingual setting recast as answer generation. These cover the script-distant languages (Japanese, Korean, Bengali, Telugu, Hindi, Arabic) where MLQA and PAWS-X show the largest transfer gaps.
The decisive check is a two-by-two over copy mechanism (present or absent) and perturbation-paired training (present or absent), read on the perturbation probe. If the copy-only model shows lower flip sensitivity than the no-copy model, copying imports the lexical-overlap shortcut into generation. That finding stands on its own and has not been shown before. If the full model restores flip sensitivity while keeping the entity gains, the method works. If entity gains vanish once the glossary-in-prompt or constrained-decoding baselines get the same renderings, the learned gate is not needed and the claim narrows accordingly. A final diagnostic tests whether the gate behaves as an overlap detector. We correlate $p_{gen}$ with the lexical overlap between each source sentence and the reference before and after perturbation training, expecting the correlation to fall.
References
- Pretrained Language Model for Text Generation: A Survey
- A Pilot Study of Domain Adaptation Effect for Neural Abstractive Summarization
- Dr. Summarize: Global Summarization of Medical Dialogue by Exploiting Local Structures
- A Copy-Augmented Generative Model for Open-Domain Question Answering
- Pointer-Generator Networks for Low-Resource Machine Translation: Don’t Copy That!
- Concept Pointer Network for Abstractive Summarization
- AMR Parsing with Action-Pointer Transformer
- Cross-lingual AMR Aligner: Paying Attention to Cross-Attention
- LargePiG: Your Large Language Model is Secretly a Pointer Generator
- PAWS: Paraphrase Adversaries from Word Scrambling
- PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification
- It Is Not Easy To Detect Paraphrases: Analysing Semantic Similarity With Antonyms and Negation Using the New SemAntoNeg Benchmark
- Semantics Altering Modifications for Evaluating Comprehension in Machine Reading
- Cross-Lingual Ability of Multilingual BERT: An Empirical Study
- Evaluating Factuality in Cross-lingual Summarization
- NCLS: Neural Cross-Lingual Summarization
- XOR QA: Cross-lingual Open-Retrieval Question Answering