Joint Encoder Decoder Pre training

15 concepts10 papers2 proposals

Early breakthroughs in self-supervised learning focused heavily on encoder-only architectures. Models learned to build rich, contextual representations of text by predicting missing words within a sentence. While this worked exceptionally well for classification or sequence labeling, it introduced a fundamental bottleneck for sequence-to-sequence (seq2seq) tasks like translation, summarization, and speech recognition.

If you couple a pre-trained, highly optimized encoder with a randomly initialized decoder, the model faces an uphill battle. The newly added decoder and, crucially, the cross-attention layers that bridge the two components have no prior understanding of how to interact. This architectural disconnect severely limits how well the model can translate its understanding into fluent generation.

To solve this, researchers shifted toward joint encoder-decoder pre-training. By optimizing the entire architecture simultaneously using sequence-reconstruction objectives, the model co-optimizes representation and generation from day one.

The Sequence to Sequence Denoising Objective

The core intuition behind joint pre-training is to make the self-supervised training task structurally identical to the downstream generation task. Instead of predicting isolated tokens, the model maps an entire corrupted input sequence to a clean target sequence.

Corrupted Input:  [The] [cat] [__] [on] [the] [__]
                         │
                  ┌──────▼──────┐
                  │   Encoder   │
                  └──────┬──────┘
                         │ Context Representations
                         ▼
                  ┌─────────────┐
                  │  Cross-Attn │ ◄─── Co-optimized during pre-training
                  └──────┬──────┘
                         │
                  ┌──────▼──────┐
                  │   Decoder   │
                  └──────▲──────┘
                         │
Target Output:     [sat] [mat]

One prominent implementation of this strategy is the Masked Sequence-to-Sequence (MASS) objective. As explored by Yingce Xia et al. and Hiroto Tamura et al., MASS masks a continuous span of tokens in the input sequence. The encoder processes the surrounding context, while the decoder is tasked with autoregressively predicting the missing fragment.

Alternatively, models like BART and mBART, utilized by Jiatao Gu et al., treat the entire architecture as a massive denoising autoencoder. The model takes a text sequence marred by noise (such as token masking, deletion, or permutation) and learns to reconstruct the original, uncorrupted document.

Why Co Optimization Works

Jointly initializing the encoder and decoder creates a powerful synergy that component-isolated pre-training cannot replicate.

1. Warming Up the Cross-Attention Bridge

In a standard Transformer seq2seq model, the cross-attention mechanism is responsible for dynamically routing information from the encoder to the decoder. As Moussa Kamal Eddine et al. demonstrate, pre-training the entire architecture end-to-end ensures that these cross-attention weights are co-optimized alongside the representation layers. This avoids the performance bottlenecks that occur when a pre-trained encoder is forced to communicate with a cold, randomly initialized decoder.

2. Establishing Structural Priors for Generation

For complex generation tasks, the decoder needs more than just a semantic understanding of the source text; it needs a structural prior for the output space. Xilun Chen et al. show that pre-trained decoders provide a significantly better initialization for generating highly structured outputs, such as nested or compositional semantic parsing trees. Similarly, Liang Wang et al. point out that joint pre-training ensures both halves of the network are aligned for text generation from the very beginning, leading to substantial gains in downstream text-to-text tasks.

Generalizing Beyond Text: Speech and Multimodality

The intuition behind joint encoder-decoder pre-training is not restricted to natural language processing; it has driven major leaps in speech processing and multimodal domains.

Traditional self-supervised learning for speech typically relies on encoder-only frameworks to extract acoustic representations from raw audio. However, Automatic Speech Recognition (ASR) is inherently a sequence-to-sequence problem. To bridge this gap, A Arunkumar et al. introduced a Transformer decoder directly into the speech self-supervised learning phase, optimizing both masked prediction and sequence prediction objectives simultaneously.

Raw Audio ──► [ Speech Encoder ] ──► Acoustic Units ──► [ Decoder ] ──► Transcribed Text
                    ▲                                         ▲
                    └─────────────── Jointly Trained ─────────┘

To train the decoder without labeled transcripts, researchers leverage synthetic targets. Frameworks like Speech2C, developed by Junyi Ao et al., tokenize continuous audio into discrete "pseudo-codes," forcing the decoder to reconstruct these codes from corrupted speech features. Similarly, Wav2Seq, proposed by Felix Wu et al., trains the model to transcribe raw audio into a synthetic "pseudo-language." This joint initialization proves incredibly potent in low-resource settings, where a randomly initialized decoder would otherwise fail to generalize from scarce labeled data.

This unified approach extends naturally to vision-and-language tasks as well. For instance, Gabriel Huang et al. successfully applied the MASS sequence-reconstruction objective to dense video captioning, demonstrating that forcing a decoder to reconstruct missing textual and audio-visual fragments together yields models that generalize far better to complex, multi-step instructional videos.

By treating understanding and generation as two halves of a single, unified problem, joint encoder-decoder pre-training ensures that no part of the network is left out in the cold.

Key Foundational Work

Microsoft Research Asia's Systems for WMT19 (2019) — contributed 2 concept elements to this direction.
Multilingual Denoising Pre-training for Neural Machine Translation (2020) — contributed 2 concept elements to this direction.
Denoising based Sequence-to-Sequence Pre-training for Text Generation (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_joint_encoder_decoder_pre_training_eefe9d10,
  title = {joint encoder-decoder pre-training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-encoder-decoder-pre-training/rdr_eefe9d10bb7f472b924acdb94e47b3fd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_joint_encoder_decoder_pre_training_eefe9d10_v1,
  title = {joint encoder-decoder pre-training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/joint-encoder-decoder-pre-training/rdr_eefe9d10bb7f472b924acdb94e47b3fd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake