In the classical formulation of sequence-to-sequence modeling, tasks are divided between two distinct entities: an encoder that digests the source sequence and a decoder that generates the target sequence. This dual-stack design, popularized by the original Transformer, relies on a specialized cross-attention mechanism to bridge the two components.
However, a growing body of research demonstrates that this architectural division is often redundant. Instead, sequence transduction can be framed as a continuous language modeling problem over a single, unified stack of Transformer layers. By concatenating the source and target sequences into a single stream, a single set of weights can handle both understanding and generation.
Traditional Dual-Stack:
[Source Sequence] ---> [ Encoder Stack ]
| (Cross-Attention Bottleneck)
v
[Target Sequence] ---> [ Decoder Stack ] ---> [Outputs]
Unified Single-Stack:
[Source Sequence + Target Sequence] ---> [ Unified Transformer Stack ] ---> [Outputs]
Dissolving the Cross-Attention Bottleneck
To understand why a single-stack architecture works, we must look at how information flows through a standard encoder-decoder model. In the dual-stack paradigm, the encoder processes the input independently of the target. The decoder then queries these static encoder representations via cross-attention. This creates an information bottleneck: the deep contextual representations learned by the encoder are only accessible through a fixed interface at the decoder layers.
A unified architecture removes this boundary entirely. By processing concatenated input segments through a single stack of identical layers, the model allows the output sequence to attend directly to the entire input sequence at every single layer. As Bae et al. observe, this configuration facilitates a much deeper contextual interaction between source and target sequences than traditional bottlenecked models can achieve.
Rather than forcing a human-designed separation between "encoding" and "matching," the unified stack allows the network to dynamically allocate its attention resources. In a case study on machine reading comprehension, Bao et al. isolated the effects of this architecture from large-scale pre-training. They discovered that a unified transformer architecture yields significant performance gains over modular pipelines purely due to its structural flexibility, learning radically different processing strategies than traditional, manually designed pipelines.
Parameter Efficiency and Continuous Representation
Beyond deeper representation learning, single-stack models offer a massive practical advantage: they drastically reduce parameter redundancy.
In a dual-stack model, the encoder and decoder maintain separate self-attention layers and feed-forward networks, effectively doubling the parameter budget. A unified approach reuses the exact same attention mechanisms and transformation layers across all stages of the sequence-to-sequence pipeline. For instance, Xia et al. demonstrated that a unified sequence module can seamlessly function as both an encoder and a decoder simply by providing a null context vector during the encoding phase, significantly lowering the total parameter count.
This architectural consolidation changes how the model treats the relationship between source and target text. Instead of viewing translation or summarization as mapping between two distinct spaces, it reformulates the task as a continuous language modeling objective. Liu et al. pioneered this for long-form text generation, showing that concatenating source documents and target summaries into a single sequence allows a decoder-only architecture to learn a single, fluid representation for both understanding context and generating text.
This continuity makes it incredibly easy to leverage pre-trained language model weights. As Song et al. note, implementing tasks like abstractive summarization within a unified decoder-only framework allows for the seamless integration of pre-trained weights across both input and output sequences, matching or exceeding the performance of dual-component systems. Similarly, the authors of DSGPT utilize special tokens to demarcate different segments within a single continuous sequence, providing a highly flexible and extensible framework for domain-specific generation.
Controlling Information Flow via Masking
If the entire sequence is fed into a single stack, how do we prevent the model from "cheating" by looking at future target tokens during training? The answer lies in the flexibility of the self-attention mask.
By manipulating the attention matrix, a single-stack architecture can emulate different behaviors:
- Prefix LM / Unified Encoder-Decoder: Tokens in the source sequence (the prefix) are allowed to attend to each other bidirectionally, while tokens in the target sequence are restricted by a causal mask to only attend to past tokens. Zhang evaluated this at scale for machine translation, finding that as models grow to billions of parameters, Prefix LMs narrow the gap with traditional encoder-decoders and even outperform them in zero-shot translation by reducing off-target errors.
- Pure Decoder-Only: The entire concatenated sequence is subjected to a strict causal mask.
- Pure Encoder-Only: The entire sequence is visible, often paired with non-autoregressive generation strategies. For example, Watanabe et al. adapted insertion-based modeling to speech recognition, proving that an encoder-only Transformer processing concatenated features can achieve competitive results without any cross-attention mechanisms.
This masking flexibility allows a single set of weights to perform multiple roles simultaneously. In document-level event extraction, Du et al. used a single set of weights to execute both encoding and decoding, ensuring the generation phase has direct, unmediated access to deep contextual features. For question generation, Varanasi et al. integrated a copy mechanism directly into a unified model's self-attention layers, maximizing computational efficiency.
Generalization Across Modalities
The intuition behind single-stack architectures extends far beyond text-to-text tasks. Because the architecture treats all inputs as a continuous sequence of tokens or vectors, it adapts naturally to speech, vision, and multi-step transmission systems.
Automatic Speech Recognition and Synthesis
In speech processing, traditional systems rely on complex, multi-component pipelines to align acoustic frames with text. Unified architectures simplify this by treating audio frames and text tokens as a single continuous sequence. Gupta et al. demonstrated that a single Transformer decoder stack can be successfully adapted for speech recognition using public corpora, eliminating the need for separate encoder and decoder components.
For speech synthesis, Du et al. introduced VALL-T, which integrates the functional components of a transducer (encoder, prediction, and joint networks) into a single Transformer decoder. This consolidation allows the model to leverage a unified context, enforcing strict monotonic alignment and drastically reducing hallucinations like skipped or repeated words.
Video and Wireless Transmission
The single-stack paradigm also excels when models must maintain state over time or handle sequential feedback loops. In video paragraph captioning, Lei et al. utilized shared Transformer layers to process video segments and sentence history into a persistent memory state, preventing context fragmentation while reducing memory overhead.
Even in wireless communication, Wu et al. applied a unified transformer architecture to image transmission. By replacing stacked modular encoders with a single model that processes both the source image and receiver channel feedback in a unified stream, their system achieves state-of-the-art reconstruction quality while remaining highly adaptable to changing channel conditions.
Conclusion
The shift toward unified, single-stack Transformer architectures represents a fundamental simplification of sequence modeling. By eliminating the artificial boundary between encoding and decoding, these architectures unlock deeper token interactions, maximize parameter utility, and provide a flexible framework that spans text, speech, and multimodal transduction.
Key Foundational Work
Generating Wikipedia by Summarizing Long Sequences (2018) — contributed 2 concept elements to this direction.
Inspecting Unification of Encoding and Matching with Transformer: A Case Study of Machine Reading Comprehension (2019) — contributed 2 concept elements to this direction.
Model-Level Dual Learning (2018) — contributed 1 concept elements to this direction.