Denoising Objectives for Sequence to Sequence Pretraining

110 concepts87 papers1 proposal

Early pre-training methods in natural language processing focused heavily on either encoder-only architectures using masked language modeling (such as BERT) or decoder-only architectures using autoregressive language modeling (such as GPT). While powerful, these objectives create a structural mismatch for sequence-to-sequence (seq2seq) tasks like translation, summarization, or paraphrasing. These tasks require mapping an arbitrary input sequence to a structurally distinct output sequence.

As noted by Liang Wang et al. in Denoising based Sequence-to-Sequence Pre-training for Text Generation, training the encoder and decoder jointly from the start ensures both components learn to coordinate for generative tasks, rather than initializing them independently. Sequence-to-sequence denoising autoencoders solve this coordination problem by introducing a unified self-supervised framework.

The Intuition of Denoising

The core idea behind a sequence-to-sequence denoising autoencoder is simple: take a clean piece of text from an unlabeled corpus, intentionally corrupt it using a specific noise function, and task the model with reconstructing the original, uncorrupted text.

[Original Text] -> [Noise Function] -> [Corrupted Input] -> [Encoder] -> [Latent Space] -> [Decoder] -> [Reconstructed Text]

Why does this build better representations? Consider an encoder that receives a sentence with missing words or scrambled syntax. To pass meaningful information to the decoder, the encoder cannot simply copy the input tokens. It must extract the underlying semantic core, effectively mapping the noisy text to a clean, invariant latent space. Meanwhile, the decoder must learn to take this latent representation and generate a fluent, grammatically correct sequence. This closely mirrors downstream generation tasks where the input (e.g., a source text in machine translation or a long article in summarization) must be transformed into a fluent target text.

Designing the Noise Function

The choice of corruption function determines exactly what the model learns. Different objectives utilize distinct forms of noise to target specific linguistic properties:

Cross-Lingual Alignment and Low-Resource Transfer

One of the most powerful emergent properties of sequence denoising is its ability to align multiple languages or modalities into a shared space. When a model is trained to denoise multiple languages simultaneously, the encoder learns to map different surface-level languages to the same underlying semantic concepts.

Yaobo Liang et al. leveraged this in XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation to build cross-lingual foundation models. This shared alignment enables zero-shot transfer: a model pre-trained via denoising can be fine-tuned on a task (such as summarization) in English, and immediately perform that task in a low-resource language without any task-specific parallel data, as shown by Zewen Chi et al. in Cross-Lingual Natural Language Generation via Pre-Training.

Beyond Text Generation: Representation Learning

While denoising autoencoders are natively generative, the act of reconstructing corrupted text forces the encoder to produce exceptionally high-quality embeddings. Because the encoder must ignore superficial noise to capture the core topical meaning, the resulting hidden states serve as excellent unsupervised representations.

For example, Silvan Wehrli et al. demonstrated in German Text Embedding Clustering Benchmark that sequential denoising autoencoders significantly refine text embeddings for downstream clustering tasks. By training the network to strip away synthetic noise, the encoder learns to map sentences based on their semantic relatedness rather than their literal token overlap.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 96 papers
  2. Version 1 · 2026-10-04 · 143 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_denoising_auto_encoding_24d0d3a0,
  title = {denoising auto-encoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/denoising-auto-encoding/rdr_24d0d3a0114a4b7aa28a5b9b0ff98ceb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_denoising_auto_encoding_24d0d3a0_v2,
  title = {denoising auto-encoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/denoising-auto-encoding/rdr_24d0d3a0114a4b7aa28a5b9b0ff98ceb/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake