Deep Encoders and Shallow Decoders for Efficient Inference

452 concepts229 papers2 proposals

For years, the default blueprint for sequence-to-sequence and image-to-image models was symmetry. If an architecture used a 6-layer Transformer encoder to digest an input, it naturally paired it with a 6-layer Transformer decoder to generate the output. This symmetry assumes that understanding an input and generating an output are equally complex tasks requiring identical computational capacity.

However, this assumption overlooks a fundamental asymmetry in how these components operate during inference. Encoders process the entire input sequence or image in parallel, leveraging highly optimized matrix multiplications. Decoders, by contrast, often operate autoregressively—generating tokens or pixels one step at a time, where each step depends on the previous one. This sequential execution turns the decoder into a severe computational bottleneck.

To break this bottleneck, a growing body of research explores asymmetric encoder-decoder depth. By shifting the parameter and layer budget heavily toward the encoder—creating a deep encoder paired with a shallow decoder—models can maintain or even improve representation quality while drastically accelerating inference.


The Autoregressive Bottleneck in Sequence Models

In tasks like neural machine translation (NMT) and speech recognition, the decoder's sequential nature means that every additional decoder layer multiplies the latency of every generated token. Early investigations into sequence model depth by Miceli Barone et al. in Deep Architectures for Neural Machine Translation revealed that while increasing encoder depth consistently yields better translations, increasing decoder depth primarily improves internal training cross-entropy without translating to better downstream evaluation metrics.

This insight paved the way for highly asymmetric production models. For instance, in Sockeye 2: A Toolkit for Neural Machine Translation, Vilar et al. demonstrated that deep-encoder, shallow-decoder configurations can cut inference time in half with negligible impacts on accuracy. Because the encoder's computation is parallelized across the entire input sequence once, adding layers to the encoder is computationally "cheap" compared to adding layers to the autoregressive decoder.

Symmetric Architecture (Slow Inference):
Input  ──> [ Deep Encoder ] ──(Parallel)──> [ Deep Decoder ] ──(Sequential)──> Output

Asymmetric Architecture (Fast Inference):
Input  ──> [ Deeper Encoder ] ──(Parallel)──> [ Shallow Decoder ] ──(Sequential)──> Output

This strategy scales remarkably well to competitive benchmarks. Hu et al. leveraged this "deep-and-shallow" paradigm in The NiuTrans System for WNGT 2020 Efficiency Task to achieve massive throughput gains.

Similarly, in speech processing, Peng et al. showed in OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer that scaling down the decoder relative to the encoder optimizes the Pareto frontier between real-time generation speed and speech-to-text accuracy. On edge hardware, where memory and compute are strictly constrained, Lv et al. confirmed in LightFormer: Light-weight Transformer Using SVD-based Weight Transfer and Parameter Sharing that asymmetric depth provides superior inference speed compared to symmetric designs.


Feature Extraction vs. Reconstruction in Vision

The utility of asymmetric depth extends far beyond text and speech generation into dense prediction tasks in computer vision, such as semantic segmentation, depth completion, and video deblurring. Here, the underlying intuition shifts slightly: the primary challenge is extracting robust, multi-scale contextual features from complex visual scenes, whereas mapping those features back to pixel-level classifications or values is computationally simpler.

In real-time semantic segmentation, models must operate at high frame rates on edge devices like autonomous vehicles. Gao et al. designed MSCFNet: A Lightweight Network With Multi-Scale Context Fusion for Real-Time Semantic Segmentation around an asymmetric structure, concentrating the computational budget on a deep, feature-rich encoder while minimizing the upsampling and reconstruction overhead in a lightweight decoder. This principle is mirrored in RGPNet: A Real-Time General Purpose Semantic Segmentation by Marzban et al., where a high-capacity encoder is paired with a minimal decoder to preserve spatial boundaries without incurring the latency of symmetric architectures.

This design assumption—that once complex features are extracted, a lightweight mapping is sufficient for pixel-wise classification—is validated across diverse imaging domains:


Forcing Semantic Compression in Representation Learning

Beyond raw inference efficiency, asymmetric depth serves as a powerful regularizer during self-supervised pre-training. When training masked autoencoders to learn dense text or image representations, a symmetric decoder can inadvertently act as a crutch. If the decoder is deep and powerful, it can reconstruct missing inputs by learning its own complex language or structural priors, allowing the encoder to get away with producing shallow, low-quality embeddings.

To combat this, Xiao et al. introduced an extreme form of asymmetry in RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder. By pairing a full-scale encoder with an intentionally "weak," single-layer decoder, the architecture creates a severe structural bottleneck.

Symmetric Autoencoder (Weak Compression):
Input ──> [ Encoder ] ──> Low-Constraint Representation ──> [ Deep Decoder ] ──> Reconstruction
                                                            (Does the heavy lifting)

RetroMAE Asymmetric Autoencoder (Strong Compression):
Input ──> [ Encoder ] ──> High-Constraint Bottleneck ──> [ 1-Layer Decoder ] ──> Reconstruction
                          (Forced to hold rich semantics)  (Too weak to help much)

Because the shallow decoder lacks the capacity to perform complex reasoning on its own, the encoder is forced to compress deep, high-fidelity semantic information into the latent embeddings to make reconstruction possible.


Why It Works: The Core Intuition

Ultimately, asymmetric encoder-decoder depth works because it aligns neural network capacity with the intrinsic difficulty of the processing phases.

In vision, feature extraction requires capturing global context, multi-scale relationships, and semantic abstractions, while reconstruction is largely a local mapping task. In language and speech, encoding can be fully parallelized across time, while autoregressive decoding scales linearly with sequence length. By breaking the assumption of architectural symmetry, deep-encoder shallow-decoder designs optimize the Pareto frontier of deep learning, delivering high-fidelity performance at a fraction of the operational cost.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 250 papers
  2. Version 1 · 2026-10-04 · 328 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_asymmetric_encoder_decoder_depth_9fa53622,
  title = {asymmetric encoder-decoder depth},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asymmetric-encoder-decoder-depth/rdr_9fa53622db6b46b39fb0c82a316d51d1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_asymmetric_encoder_decoder_depth_9fa53622_v2,
  title = {asymmetric encoder-decoder depth},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/asymmetric-encoder-decoder-depth/rdr_9fa53622db6b46b39fb0c82a316d51d1/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake