Sequence to Sequence Denoising for Natural Language Generation

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2143 papers153 conceptsfrozen 2026-10-04

In the early days of large-scale pre-training, the field was split between two paradigms: encoders like BERT, which excel at understanding context by looking at both sides of a word, and decoders like GPT, which excel at generating text by predicting the next token. However, many of the most important tasks in NLP—summarization, translation, and grammatical correction—are neither purely understanding nor purely generation. They are transformations.

The research direction of sequence-to-sequence (seq2seq) denoising bridges this gap. By intentionally corrupting a text sequence and training a model to reconstruct the original, researchers have created "foundation models" that are natively designed for transformation tasks.

The Intuition of Learning through Reconstruction

The core idea is simple: if a model can perfectly reconstruct a document that has been shuffled, masked, and partially deleted, it must understand both the local syntax and the global structure of the language.

As Mike Lewis, Omer Levy, and their colleagues demonstrated with BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, this "denoising" process is more flexible than standard language modeling. Unlike BERT, which predicts individual missing words, a denoising autoencoder can handle "noise functions" that change the length of the document, such as deleting entire spans of text or permuting the order of sentences. This forces the model to learn not just what word comes next, but how a coherent narrative is assembled.

Span Corruption as a Universal Objective

A recurring mechanism across this research is span corruption. Instead of masking random individual tokens, the model masks contiguous segments of text. Adam Roberts, Hyung Won Chung, and others found in their systematic study of architectures that this objective, when paired with an encoder-decoder architecture (like T5), provides a superior foundation for zero-shot generalization compared to standard causal modeling.

This approach has proven remarkably robust across different domains:

  • Programming: Ahmed Elnaggar and the CodeTrans team applied span corruption to 38 million instances of code, allowing the model to learn the underlying logic and syntax of software.
  • Biomedicine: Hongyi Yuan developed BioBART by using text-infilling on PubMed abstracts, enabling the model to master technical medical vocabulary.
  • Multilingualism: Raj Dabre and Anoop Kunchukuttan used masked span reconstruction in IndicBART to unify 11 Indic languages into a single representational space.

Architecture Agnosticism

While the Transformer is the dominant architecture for these tasks, the denoising principle itself is what drives the performance. Mostafa Dehghani and Yi Tay challenged the "unbreakable tie" between Transformers and pre-training in Are Pre-trained Convolutions Better than Pre-trained Transformers?. They showed that when modern denoising objectives like span-based denoising are applied to Convolutional Neural Networks (CNNs), the CNNs can actually outperform Transformers on many NLP tasks while being more computationally efficient.

Unifying the Paradigms

The field is currently moving toward a unification of these denoising strategies. Yi Tay and Vinh Q. Tran introduced UL2: Unifying Language Learning Paradigms, which uses a Mixture-of-Denoisers (MoD). This framework combines "R-denoising" (regular span corruption), "S-denoising" (sequential reconstruction), and "X-denoising" (extreme span corruption).

The intuition behind MoD is that different scales of noise teach the model different skills. Small gaps teach local grammar, while extreme gaps—where the model must recover 50% of a document—force the model to act like a generative engine, bridging the gap between the "understanding" capabilities of T5 and the "creative" capabilities of GPT-3.

Why It Works: The Latent Bridge

Ultimately, seq2seq denoising works because it forces the encoder and decoder to coordinate. In a standard language model, the decoder only looks at its own previous words. In a denoising autoencoder, as Liang Wang argues in PoDA, the model must learn to map a "noisy" input into a clean "latent representation" that the decoder can then expand back into fluent text. This latent bridge is what makes these models so effective for downstream tasks like summarization and translation, where the goal is to extract the "essence" of an input and reconstruct it in a new form.

Papers in this version

Showing 60 of 143 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 96 papers
  2. Version 1 · 2026-10-04 · 143 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_denoising_auto_encoding_24d0d3a0,
  title = {denoising auto-encoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/denoising-auto-encoding/rdr_24d0d3a0114a4b7aa28a5b9b0ff98ceb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_sequence_to_sequence_denoising_for_natur_24d0d3a0_v1,
  title = {Sequence to Sequence Denoising for Natural Language Generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sequence-to-sequence-denoising-for-natural-language-generation/rdr_24d0d3a0114a4b7aa28a5b9b0ff98ceb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}