Breaking the Identity Mapping Trap in Neural Sequence Editing

This direction is merged into another direction: identity mapping bias. Its last version stays readable and citable here.

Version 1 of 159 papers75 conceptsfrozen 2026-10-04

In tasks like grammatical error correction (GEC), text simplification, and code editing, the input and output sequences are often 90% identical. While this high overlap seems like it should make the task easier, it actually creates a fundamental failure mode for neural networks: the copying trap.

When a model is trained to minimize loss on sequences that are nearly identical, the "statistically safest" strategy is to simply copy the input verbatim. This identity mapping is a powerful local optimum that prevents models from performing the creative rewrites or structural changes they were designed for.

The Statistical Safety of Laziness

The core of the problem lies in the training data distribution. As Marcin Junczys-Dowmunt and colleagues noted in their work on Neural GEC, neural models often lag behind traditional methods because they over-rely on copying. Because the identity mapping is the most frequent pattern in the training data, the model perceives any change as a high-risk move that could increase training error.

Xingxing Zhang and Mirella Lapata describe this as "copy-paste laziness." In their DRESS model, they argue that standard sequence-to-sequence models default to copying because it is the most reliable way to maintain high scores in similarity-based metrics like BLEU, even if the task—such as simplification—requires radical structural reorganization.

The Pre-training Paradox

Modern language models are often pre-trained on reconstruction tasks (like span masking in T5 or BART), which explicitly rewards the model for recovering the original input. While this builds a strong understanding of language, it also reinforces the identity bias.

This becomes particularly problematic in unsupervised tasks. Wasi Uddin Ahmad and Baishakhi Ray found that models pre-trained on code reconstruction struggle with unsupervised code translation because they are "too good" at reconstructing the source language. Instead of translating Java to Python, the model simply outputs a slightly modified version of the Java source, unable to cross the "language barrier" due to its ingrained reconstruction bias.

Forcing the Model to Edit

To break this trap, researchers have moved away from "black-box" translation toward explicit editing frameworks. Instead of generating a whole new sentence, models like EditNTS, developed by Yue Dong, use a Neural Programmer-Interpreter to predict specific operations: KEEP, DELETE, or ADD. By forcing the model to choose an action for every token, the "default" of doing nothing becomes an explicit choice (KEEP) rather than a passive failure of the decoder.

Another approach involves changing the training objective itself. Kai Shen and Yichong Leng introduced MaskCorrect, which masks out correct tokens during training. This forces the model to reconstruct the text from context rather than relying on the input tokens as a crutch, effectively breaking the trivial identity function.

Decoding with Constraints

If the model is "lazy" during inference, researchers can use decoding constraints to force its hand. Tomoyuki Kajiwara proposed negative lexically constrained decoding, which forbids the model from outputting specific complex words from the source. This forces the model to find synonyms, overcoming the conservative tendency to retain technical jargon.

Similarly, in medical text simplification, Lorenzo Jaime Yu Flores and Arman Cohan utilized unlikelihood training. By explicitly penalizing the model for copying complex technical terms, they ensure the output actually bridges the communication gap rather than just mirroring the source's complexity.

Structural Decomposition as a Pre-emptive Strike

Sometimes the identity bias is so strong because the sentence structure itself is a "trap." Elior Sulem and Omri Abend found that by splitting complex sentences into simpler semantic "scenes" before passing them to a neural model, they could reduce the identity rate from over 50% to nearly zero.

When the input is already structurally simplified, the neural model can no longer "hide" behind the original complex syntax. This pre-processing step, also explored in the DisSim framework by Christina Niklaus, transforms the task from a difficult structural rewrite into a simpler lexical paraphrasing task, effectively bypassing the model's conservative instincts.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_breaking_the_identity_mapping_trap_in_ne_7a5835cc,
  title = {Breaking the Identity Mapping Trap in Neural Sequence Editing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/breaking-the-identity-mapping-trap-in-neural-sequence-editing/rdr_7a5835cc7111452eba9facf971026f9e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_breaking_the_identity_mapping_trap_in_ne_7a5835cc_v1,
  title = {Breaking the Identity Mapping Trap in Neural Sequence Editing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/breaking-the-identity-mapping-trap-in-neural-sequence-editing/rdr_7a5835cc7111452eba9facf971026f9e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}