Deep learning models are fundamentally opportunistic: they will always exploit the easiest statistical shortcut to minimize training loss. In many sequence-to-sequence (Seq2Seq) and text generation tasks, this opportunism manifests as a "lazy" copying behavior.
Consider tasks like grammatical error correction (GEC) or spelling correction. Because errors are sparse, the vast majority of tokens in the input sequence should remain unchanged in the output. A model can achieve remarkably low loss simply by learning an identity mapping—blindly copying tokens from the input to the output without ever understanding the underlying linguistic structure.
This "tunnel vision" phenomenon, as described by Zeliang Song et al. in Image Captioning with Context-Aware Auxiliary Guidance, occurs when models predict tokens based primarily on local token matching or immediate prefix histories rather than broader context. As noted in a survey by Junyi Li et al. (Pretrained Language Model for Text Generation: A Survey), while decoder-only architectures frequently adapt to sequence tasks by concatenating inputs and outputs, standard causal masking alone does not prevent the model from over-relying on local shortcuts. When models can minimize loss via simple copying, they fail to build robust representations, leading to text degeneration, hallucinations, or a failure to adhere to provided instructions.
The Core Mechanism: Breaking the Identity Mapping
To force a model to actually learn language dependencies and context, a powerful strategy has emerged across the literature: deliberately masking correct or target tokens during training.
If the shortcut is to copy a token from the input or read it from the immediate prefix history, the solution is to remove that token from the model's view. By replacing valid, correct tokens with a mask symbol ([MASK]), the model can no longer rely on a trivial identity mapping. To minimize loss, it is forced to route information through its attention layers, reconstructing the missing token entirely from the surrounding semantic context.
Standard Training (Lazy Shortcut):
Input: The sky is blue .
Output: The sky is blue . <-- Model just copies "blue" without understanding
Correct-Token Masking (Forced Grounding):
Input: The sky is [MASK] .
Output: The sky is blue . <-- Model must use "sky" and "is" to deduce "blue"
Why It Works: Shifting to Active Reconstruction
To understand why this works, consider how information flows through a transformer network. When a model copies an input token, the attention weights can simply focus heavily on the corresponding position in the source sequence. The network acts as a passive pass-through.
When we apply a strategy like MaskCorrect, proposed by Kai Shen et al. in Mask the Correct Tokens: An Embarrassingly Simple Approach for Error Correction, a percentage of tokens that already match the target sequence are intentionally hidden. Suddenly, the passive pass-through fails. The model must now look at the rest of the sentence to deduce what word fits the grammatical and semantic profile of the masked slot. This shifts the training objective from simple sequence mapping to active language modeling.
Similarly, in Chinese Spelling Correction (CSC), Hongqiu Wu et al. (Rethinking Masked Language Modeling for Chinese Spelling Correction) introduced Masked Fine-Tuning (Masked-FT) to prevent models from merely memorizing common typo patterns. By masking correct characters, the model learns to evaluate the validity of characters based on context, allowing it to generalize to unseen errors during inference. Linfeng Liu et al. (Chinese Spelling Correction as Rephrasing Language Model) extended this concept by framing spelling correction as a rephrasing task, using auxiliary masking on correct characters to break positional alignments and force a holistic semantic understanding.
Applications Across the Generation Pipeline
This intuition—that masking target information forces deeper contextual grounding—spans across pre-training, fine-tuning, and in-context learning.
Pre-training and Architecture Coordination
In encoder-decoder frameworks, models can struggle to coordinate effectively if the decoder relies too heavily on its own local generation history. The MASS framework, developed by Kaitao Song et al. (MASS: Masked Sequence to Sequence Pre-training for Language Generation), masks non-target tokens in the decoder during pre-training. This forces the decoder to extract information from the encoder's latent representations rather than relying on local token matching.
For sequence-to-sequence pre-training, Song Xu 0002 et al. introduced prefix-masked decoding in their framework, PRINCE: Prefix-Masked Decoding for Knowledge Enhanced Sequence-to-Sequence Pre-Training. By replacing ground-truth tokens in the decoder's history with mask symbols, they prevent the model from relying on immediate local linguistic associations, compelling it to utilize the encoder's context to generate factual entities.
Mitigating Copying in Error Correction and Question Generation
When generating text from a source passage, copying can be detrimental. In Neural Question Generation, Hwanhee Lee et al. (Improving Neural Question Generation Using Answer Separation) separated the answer from the passage by replacing target answer tokens with a placeholder. This prevented the model from accidentally copying the answer into the generated question, forcing it to use surrounding context clues to determine the proper interrogative structure.
For unsupervised grammatical error correction, Jiayi Deng et al. (InstructGEC: Enhancing Unsupervised Grammatical Error Correction with Instruction Tuning) utilized context-forced masking to hide tokens in the source sentence, ensuring the model relies on edit instructions and context rather than falling back on an identity mapping. To provide a pure linguistic signal during spelling correction training, Chi Wei et al. (Training a Better Chinese Spelling Correction Model via Prior-knowledge Guided Teacher) masked error positions in the input to a teacher network, forcing it to generate corrections based entirely on context.
Enhancing Autoregressive Grounding and Reducing Hallucinations
Even in modern decoder-only Large Language Models (LLMs), copying shortcuts can degrade performance during supervised fine-tuning (SFT). Xialie Zhuang et al. (Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More) introduced Mask-Enhanced Autoregressive Prediction (MEAP), where target response tokens are randomly hidden during SFT. Calculating loss primarily on these masked tokens compels the LLM to ground its responses deeply in the provided context, significantly reducing hallucinations.
Similarly, to ensure models adhere to explicit constraints during multi-task machine translation, Chunyou Li et al. (MT2: Towards a Multi-Task Machine Translation Model with Translation-Specific In-Context Learning) targeted tokens in the target sequence that appeared in the input context for masking. This forced the architecture to actively retrieve and adapt information from the prompt rather than relying on internal weights.
Finally, this mechanism can even combat text degeneration. Huayang Li et al. (Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective) introduced Repetition Dropout, which randomly masks attention to repetitive tokens during training. This forces the model to predict subsequent words based on broader semantic context rather than getting trapped in local, repetitive loops.
Conclusion
Masking correct or target tokens turns a passive sequence-mapping task into an active, context-driven reconstruction challenge. By intentionally breaking identity mappings and blocking local shortcuts, this family of techniques forces neural networks to build robust functional relationships with their context. The result is a class of models that generalize better, hallucinate less, and truly utilize the text they process.
Key Foundational Work
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019) — contributed 2 concept elements to this direction.
Image Captioning with Context-Aware Auxiliary Guidance (2021) — contributed 1 concept elements to this direction.
Improving Neural Question Generation Using Answer Separation (2019) — contributed 1 concept elements to this direction.