The Decoder Initialization Gap

22 concepts18 papers1 proposal

When building sequence-to-sequence or dense prediction models, a common and powerful recipe is to use a large-scale, self-supervised model as an encoder. Whether it is a text encoder like BERT, a vision transformer, or a self-supervised speech backbone, these models contain rich, mature representations of their respective domains.

However, when we adapt these pre-trained encoders to generative or dense prediction tasks—such as text summarization, image segmentation, or speech synthesis—we must attach a decoder component. If this decoder is initialized randomly from scratch, an architectural and optimization imbalance occurs. This phenomenon is known as the decoder initialization gap.

Pairing a highly mature encoder with an un-pretrained decoder introduces severe training instabilities, optimization mismatches, and data inefficiencies that can undermine the very benefits of pre-training.


The Maturity Mismatch

The core intuition behind the decoder initialization gap lies in a fundamental discrepancy in component maturity. When fine-tuning begins, the encoder already possesses a sophisticated understanding of structural and contextual features, while the decoder is entirely unoptimized.

As observed by Mirella Lapata et al. in Text Summarization with Pretrained Encoders, this discrepancy causes severe training instability. Because the encoder is already highly capable, it can easily overfit the training data before the decoder has acquired the basic linguistic competence needed to generate coherent text.

Furthermore, an un-pretrained decoder lacks crucial structural priors. Xilun Chen et al. demonstrate in Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing that this deficiency restricts the model's ability to produce highly structured outputs, especially in low-resource settings where data is too scarce for the decoder to learn these priors from scratch.


Why It Fails: The Mechanics of Optimization Mismatch

The decoder initialization gap is not merely a matter of the decoder needing time to "catch up." The asymmetry actively harms the network through two primary optimization failures:

1. Gradient Toxicity and Representation Destruction

During the initial phases of backpropagation, the randomly initialized decoder outputs chaotic, high-variance predictions. As a result, it generates highly erratic gradients. When these gradients flow backward through the network, they risk destroying the sophisticated, fragile features already learned by the pre-trained encoder.

Yawei Kong et al. document this phenomenon in Probing Multi-modal Machine Translation with Pre-trained Language Model, showing that early training instability can actively degrade encoder priors. Similarly, the authors of Court Opinion Generation from Case Fact Description with Legal Basis describe this as an optimization mismatch where the encoder suffers from catastrophic forgetting before the decoder can even begin to converge on the task. This representational mismatch can manifest as sudden, transient drops in reconstruction quality early in the optimization process, as noted by Kushal Kardam Vyas et al. in Learning Transferable Features for Implicit Neural Representations.

2. The Capacity and Data Bottleneck

When a high-capacity pre-trained encoder is paired with a small, uninitialized decoder, a severe capacity mismatch occurs. Gözde Gül Şahin et al. show in PuzzLing Machines: A Challenge on Learning From Small Data that this imbalance destabilizes learning when local task data is limited, proving that massive encoder priors do not translate to downstream success if the decoder cannot process them.

This bottleneck is highly visible in dense prediction and cross-modal tasks:


Strategies for Rebalancing the Architecture

To overcome the decoder initialization gap, researchers have developed paradigms aimed at balancing the maturity and workloads of both components.

Joint Sequence-to-Sequence Pre-training

The most direct solution is to pre-train the encoder and decoder simultaneously. Liang Wang et al. (Denoising based Sequence-to-Sequence Pre-training for Text Generation) introduced early frameworks designed to optimize simultaneously for bidirectional context representation and unidirectional sequence generation. By pre-training the entire generative stack on structured objectives, as argued by Gary D. Bader et al. (A sequence-to-sequence approach for document-level relation extraction), the decoder develops a native ability to map raw representations to structured schemas before fine-tuning even begins.

Targeted Decoder Pre-training

When the encoder's weights are already fixed, researchers introduce self-supervised objectives exclusively for the decoder. For instance, the framework proposed by Emmanuel Asiedu Brempong et al. (Decoder Denoising Pretraining for Semantic Segmentation) pre-trains the decoder to remove noise from images. This teaches the decoder spatial reconstruction and upsampling prior to fine-tuning, yielding massive performance gains in few-shot scenarios.

Workload and Capacity Reallocation

An alternative perspective involves rebalancing how much work each component performs during reconstruction tasks. Ran Ma et al. (Reconstruction Target Matters in Masked Image Modeling for Cross-Domain Few-Shot Learning) demonstrate that an overly powerful decoder can inadvertently over-compensate for a weak encoder by doing too much heavy lifting during reconstruction. They propose using a minimalist, lightweight decoder, which forces the encoder to produce highly robust, transferable features that generalize better to downstream tasks.

Key Foundational Work

PuzzLing Machines: A Challenge on Learning From Small Data (2020) — contributed 3 concept elements to this direction.
Text Summarization with Pretrained Encoders (2019) — contributed 2 concept elements to this direction.
Low-Resource Domain Adaptation for Compositional Task-Oriented Semantic Parsing (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_decoder_initialization_gap_741f2644,
  title = {decoder initialization gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoder-initialization-gap/rdr_741f264466424f20a6dc3e033b2eb732},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_decoder_initialization_gap_741f2644_v1,
  title = {decoder initialization gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decoder-initialization-gap/rdr_741f264466424f20a6dc3e033b2eb732/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake