The Identity Mapping Bias in Sequence Transformation

71 concepts55 papers2 proposals

Imagine asking a language model to fix a single typo in a paragraph or simplify a complex sentence. Intuitively, the model should identify the flaw, compute the alternative phrasing, and output the corrected text. In practice, however, sequence-to-sequence models frequently fall into a "copying trap." Instead of executing the requested edit, they reproduce the input sequence verbatim.

This phenomenon is known as the identity mapping bias or copying bias. It represents a fundamental challenge across text editing, simplification, grammatical error correction, and program repair.

The Root of the Bias: High Sequence Overlap

Why do highly capable models default to doing nothing? The answer lies in the statistical distribution of sequence transformation datasets.

In tasks like grammatical error correction (GEC), sentence simplification, and code bug-fixing, the source and target sequences share massive token overlap. If 95% of the tokens in a sentence are already correct, a model can achieve exceptionally low training error simply by learning a trivial identity function—copying every token from input to output. As Kriz et al. observe, because keeping words unchanged is the most frequent operation in training data, identity mapping becomes a powerful local optimum during optimization.

Source: [The] [cat] [sat] [on] [the] [mat] [and] [it] [was] [very] [fat]
                                                                │ (Target: delete "very")
Target: [The] [cat] [sat] [on] [the] [mat] [and] [it] [was]         [fat]
        └───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┘
                     90% of tokens are identical!

When a model is trained on thousands of examples where the vast majority of tokens remain unchanged, the attention mechanisms and token probabilities become heavily weighted toward replication. Consequently, the model struggles to muster the statistical "confidence" required to deviate from the source text when an actual edit is necessary.

How the Copying Trap Manifests Across Domains

The identity mapping bias is not unique to a single architecture or task; it consistently emerges wherever input and output sequences closely resemble one another.

Text Simplification

When evaluating language models on sentence simplification, Kew et al. found that open-weight models frequently prioritize meaning preservation by copying the source text verbatim rather than performing actual simplifications like rephrasing or sentence splitting. This conservative behavior allows models to score highly on surface-level fidelity metrics while failing to make the text more accessible. Similarly, Cripwell notes that this excessive conservatism stems directly from training on datasets containing many identical or near-identical sentence pairs, teaching the model to treat the identity transformation as the safest bet.

Error Correction and Code Repair

In tasks where errors are sparse, the identity mapping bias severely degrades performance. Shen et al. explain that when errors are rare, models "lazily" learn to copy input tokens rather than understanding underlying language context.

This creates a severe confidence imbalance. As Cao et al. point out, autoregressive models develop overconfidence in identity mappings, leaving them highly underconfident at the exact positions where specific edits (like substitutions or deletions) are required. The same issue plagues automated program repair; Liu et al. demonstrate that high similarity between buggy and fixed code causes models to replicate the buggy input rather than identifying and applying the necessary logical changes.

Unsupervised Translation and Paraphrasing

The bias even disrupts cross-lingual and stylistic generation. Ahmad shows that sequence-to-sequence models pre-trained on code reconstruction struggle with unsupervised translation because their pre-training objectives heavily incentivize input reconstruction, causing them to output the source language even when prompted for a target language. In monolingual paraphrasing, Niu et al. find that models optimized for text reconstruction default to exact sequence replication due to a lack of explicit incentives to alter surface structure.

Breaking the Copying Trap

To overcome this bias, researchers have developed strategies that prevent copying from acting as a statistical shortcut. These solutions generally fall into three categories: altering the training objective, forcing explicit edit plans, and using auxiliary guidance.

1. Strategic Masking (Objective Tweaks)

One way to break the identity mapping bias is to alter the training objective so that copying is no longer a viable shortcut. Shen et al. introduced MaskCorrect, a method that masks a portion of the correct tokens during training. By forcing the model to actively reconstruct correct tokens from context rather than just passing them through from the input, the model is broken out of its lazy copying habits and forced to learn deeper linguistic dependencies.

2. Explicit Edit Plans

Instead of generating the entire sequence from scratch—which Kaneko et al. note is computationally expensive and highly prone to copy bias—models can be designed to output explicit editing instructions.

Standard Seq2Seq:  [Input Text] ───────────────────────────> [Complete Output Text] (High Copy Bias)

Edit-Based Model:  [Input Text] ─> [Plan: KEEP, DELETE, ADD] ─> [Applied Edits Only]  (Low Copy Bias)

For instance, Dong et al. developed EditNTS, a neural programmer-interpreter that explicitly predicts KEEP, DELETE, and ADD operations. By shifting the task from translation to explicit editing, the model bypasses the implicit copying bias. Similarly, the CoditT5 framework proposed by Zhang et al. requires the model to generate an explicit structural "edit plan" before producing the final text, ensuring that modifications are intentionally structured rather than accidentally omitted.

3. Auxiliary Guidance

Another strategy involves using secondary models to counteract autoregressive overconfidence. Cao et al. introduced a joint training framework where a non-autoregressive model acts as a global auxiliary. This auxiliary model identifies edit positions ahead of time, helping the primary autoregressive model focus its attention specifically on the tricky positions that require actual modifications, effectively neutralizing the identity mapping trap.

Key Foundational Work

Identifying targets for syntactic simplification (2011) — contributed 1 concept elements to this direction.
Unlabeled Data for Morphological Generation With Character-Based Sequence-to-Sequence Models (2017) — contributed 1 concept elements to this direction.
CoNLL-SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection in 52 Languages (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_identity_mapping_bias_6136e379,
  title = {identity mapping bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/identity-mapping-bias/rdr_6136e379c23743c5a6658168c62ac037},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_identity_mapping_bias_6136e379_v1,
  title = {identity mapping bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/identity-mapping-bias/rdr_6136e379c23743c5a6658168c62ac037/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake