Early pre-training methods in natural language processing focused heavily on either encoder-only architectures using masked language modeling (such as BERT) or decoder-only architectures using autoregressive language modeling (such as GPT). While powerful, these objectives create a structural mismatch for sequence-to-sequence (seq2seq) tasks like translation, summarization, or paraphrasing. These tasks require mapping an arbitrary input sequence to a structurally distinct output sequence.
As noted by Liang Wang et al. in Denoising based Sequence-to-Sequence Pre-training for Text Generation, training the encoder and decoder jointly from the start ensures both components learn to coordinate for generative tasks, rather than initializing them independently. Sequence-to-sequence denoising autoencoders solve this coordination problem by introducing a unified self-supervised framework.
The Intuition of Denoising
The core idea behind a sequence-to-sequence denoising autoencoder is simple: take a clean piece of text from an unlabeled corpus, intentionally corrupt it using a specific noise function, and task the model with reconstructing the original, uncorrupted text.
[Original Text] -> [Noise Function] -> [Corrupted Input] -> [Encoder] -> [Latent Space] -> [Decoder] -> [Reconstructed Text]
Why does this build better representations? Consider an encoder that receives a sentence with missing words or scrambled syntax. To pass meaningful information to the decoder, the encoder cannot simply copy the input tokens. It must extract the underlying semantic core, effectively mapping the noisy text to a clean, invariant latent space. Meanwhile, the decoder must learn to take this latent representation and generate a fluent, grammatically correct sequence. This closely mirrors downstream generation tasks where the input (e.g., a source text in machine translation or a long article in summarization) must be transformed into a fluent target text.
Designing the Noise Function
The choice of corruption function determines exactly what the model learns. Different objectives utilize distinct forms of noise to target specific linguistic properties:
- Span Infilling and Masking: In Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, Colin Raffel et al. replaced contiguous sequences of tokens with unique sentinel tokens, training the model to predict only the missing spans. Similarly, Mike Lewis et al. introduced BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension, which decoupled input and output lengths by deleting tokens entirely. This forces the model to decide how many tokens are missing, teaching it local syntax and vocabulary density.
- Sentence Permutation and Word Shuffling: To learn global document structure and discourse coherence, models are tasked with reordering text. Felix Hill et al. demonstrated in Learning Distributed Representations of Sentences from Unlabelled Data that random word deletions and swaps allow models to capture robust semantic representations without relying on a coherent narrative structure. In unsupervised translation, Mikel Artetxe et al. used word swapping in Unsupervised Neural Machine Translation to prevent the model from learning simple identity mappings, forcing it to internalize target language syntax.
- Domain-Specific Noise: The denoising framework easily adapts to non-standard text domains. For instance, Baptiste Roziere applied token shuffling and deletion to source code in Unsupervised Translation of Programming Languages, compelling the model to learn the strict logical and syntactic constraints of programming languages like C++, Java, and Python. In spoken language processing, Qianqian Dong et al. injected artificial repetitions and filler words in Adapting Translation Models for Transcript Disfluency Detection to teach models how to clean conversational transcripts.
Cross-Lingual Alignment and Low-Resource Transfer
One of the most powerful emergent properties of sequence denoising is its ability to align multiple languages or modalities into a shared space. When a model is trained to denoise multiple languages simultaneously, the encoder learns to map different surface-level languages to the same underlying semantic concepts.
Yaobo Liang et al. leveraged this in XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation to build cross-lingual foundation models. This shared alignment enables zero-shot transfer: a model pre-trained via denoising can be fine-tuned on a task (such as summarization) in English, and immediately perform that task in a low-resource language without any task-specific parallel data, as shown by Zewen Chi et al. in Cross-Lingual Natural Language Generation via Pre-Training.
Beyond Text Generation: Representation Learning
While denoising autoencoders are natively generative, the act of reconstructing corrupted text forces the encoder to produce exceptionally high-quality embeddings. Because the encoder must ignore superficial noise to capture the core topical meaning, the resulting hidden states serve as excellent unsupervised representations.
For example, Silvan Wehrli et al. demonstrated in German Text Embedding Clustering Benchmark that sequential denoising autoencoders significantly refine text embeddings for downstream clustering tasks. By training the network to strip away synthetic noise, the encoder learns to map sentences based on their semantic relatedness rather than their literal token overlap.