Preventing Copying in Sequence Generation

This direction is merged into another direction: correct-token masking. Its last version stays readable and citable here.

Version 1 of 112 papers16 conceptsfrozen 2026-10-04

In sequence-to-sequence (Seq2Seq) models, a recurring failure mode is the "copying shortcut." When a model is tasked with generating an output based on a source—whether translating a sentence, correcting a typo, or generating a question—it often learns to bypass complex reasoning by simply copying tokens directly from the input or its own recent history. Recent research has converged on a suite of architectural and training strategies designed to break this dependency, forcing models to rely on latent context and semantic understanding instead of literal token matching.

The Problem of the Identity Shortcut

The core challenge in many generation tasks is that the target output often shares significant overlap with the source input. In tasks like grammatical error correction or document-level translation, the model may find that the easiest way to minimize loss is to act as an identity function. Kai Shen and Xu Tan describe this as "lazy" learning: because errors are sparse, the model defaults to copying the input, which prevents it from actually learning the underlying linguistic rules required to fix the few errors that do exist.

Similarly, in Neural Question Generation, models often erroneously include the answer within the question itself. Hwanhee Lee and Kyomin Jung found that by explicitly masking the answer in the source passage and processing it through a separate "keyword-net," they could force the model to use the surrounding context to determine the proper interrogative structure without "leaking" the answer into the output.

Masking to Force Latent Reasoning

To combat these shortcuts, researchers have introduced various forms of intentional information bottlenecks. The intuition is simple: if the model cannot see the "easy" answer, it must learn to infer it from the "hard" context.

  • Source-Side Masking: In the MaskCorrect approach, Shen et al. intentionally mask correct tokens in the source. This forces the model to reconstruct those tokens from context, ensuring it learns language modeling rather than just copying.
  • Decoder-Side Masking: During pre-training, the MASS framework by Kaitao Song and Xu Tan masks tokens in the decoder that are not part of the current target fragment. This prevents the decoder from matching tokens directly to the encoder and forces it to extract meaning from the encoder's latent representations.
  • Prefix Masking: To improve factual accuracy, the PRINCE framework by Song Xu and Xiaodong He replaces the ground-truth tokens in the decoder's history with mask symbols. This stops the model from relying on local linguistic patterns (like "The capital of France is...") and forces it to consult the encoder's knowledge to generate the next token.

Selective Attention as a Filter

Another facet of this research direction involves restricting the "view" of the decoder. In document-level translation, providing too much context can overwhelm the model. Liangyou Li and Qun Liu demonstrated that translation quality improves when the decoder is restricted from attending directly to the extended context. Instead, the encoder processes the broad context via self-attention, and the decoder only attends to the final, context-aware representations of the source sentence.

This "selective cross-attention" is also utilized in dialogue translation by Sumire Honda and Patrick Fernandes. By preventing the decoder from attending to historical context tokens directly, the model avoids being distracted by irrelevant data while still benefiting from the "big picture" synthesized by the encoder.

Summary of Intuition

The shared goal across these papers is to move beyond surface-level pattern matching. By masking target information in the source, masking history in the decoder, or restricting attention to synthesized representations, these mechanisms ensure that the model's internal "path of least resistance" must pass through a deep understanding of the context. This results in generation that is more logical, factually grounded, and less prone to the repetitive errors of simple sequence-to-sequence mapping.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_preventing_copying_in_sequence_generatio_e47ccaa5,
  title = {Preventing Copying in Sequence Generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preventing-copying-in-sequence-generation/rdr_e47ccaa572df4002824dbc8447cf48e5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_preventing_copying_in_sequence_generatio_e47ccaa5_v1,
  title = {Preventing Copying in Sequence Generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preventing-copying-in-sequence-generation/rdr_e47ccaa572df4002824dbc8447cf48e5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}