Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder

Zihao Fu, Wai Lam, Qian Yu, Anthony Man-Cho So, Shengding Hu, Zhiyuan Liu, Nigel CollierarXiv 2023 · DOI · arXiv

Abstract

While decoder-only language models (LMs) like GPT have become the standard for large-scale pre-training, they often underperform traditional encoder-decoder (ED) architectures on sequence-to-sequence (seq2seq) tasks. This paper investigates this performance gap by introducing the Regularized Encoder-Decoder (RED) framework, which interprets LMs as ED models subject to specific structural regularizations. The authors identify the Attention Degeneration Problem (ADP)—a mathematical decay in sensitivity to source inputs as generation length increases—and propose the Partial Attention Language Model (PALM) to mitigate these issues. Experimental results across translation and summarization benchmarks demonstrate that PALM achieves superior performance and grounding compared to standard LMs.

Introduction

In recent years, the Natural Language Processing landscape has shifted from specialized Encoder-Decoder (ED) models (e.g., T5, BART) toward unified Decoder-Only Language Models (LM) (e.g., GPT-4, Llama). While LMs are easier to scale and pre-train, they frequently struggle with seq2seq tasks like machine translation and summarization, often suffering from hallucinations or premature stopping. This research seeks to understand the structural reasons behind this performance gap by asking: What is the mathematical difference between an LM and an ED model, and how does that difference affect performance?

Conceptual comparison between traditional Encoder-Decoder architectures and Decoder-Only Language Models for translation tasks.

The RED Framework and Attention Degeneration

To bridge the gap between these architectures, the authors propose the Regularized Encoder-Decoder (RED) framework. RED is a theoretical construct that configures an ED model to behave exactly like a decoder-only LM. This allows researchers to isolate specific "regularizations" inherent in LMs, such as parameter sharing between the encoder and decoder components and the use of consecutive positional encoding across the entire sequence.

Through sensitivity analysis using Jacobian matrices, the authors mathematically prove the existence of the Attention Degeneration Problem (ADP). They demonstrate that in a standard LM, the output's sensitivity to the source input decreases as the generation step increases. Essentially, as the model generates more tokens, the "attention budget" for the original source text is diluted by the growing target sequence, causing the model to "forget" the prompt.

The RED framework (right) interprets the Language Model as an Encoder-Decoder (left) with specific regularizations like parameter sharing and consecutive positional encoding.

PALM: The Partial Attention Language Model

To address ADP, the authors introduce the Partial Attention Language Model (PALM). PALM incorporates three key structural modifications:

  1. Partial Attention (PA): A dedicated attention component that focuses exclusively on the source sequence, ensuring the source signal remains strong regardless of output length.
  2. Bidirectional Source Attention: Unlike the causal masking in standard LMs, PALM allows tokens in the source sequence to attend to each other bidirectionally.
  3. Positional Restart: PALM restarts positional encodings for the target sequence, helping the model better distinguish between the source and target domains.

Experimental Results

The researchers evaluated PALM across standard benchmarks including IWSLT’14, WMT’16, and XSUM. Key findings include:

  • Efficiency and Accuracy: PALM consistently outperformed standard LMs. Notably, a PALM model with 35.6M parameters often outperformed traditional ED models with 47.1M parameters on translation tasks.
  • Reduced Hallucinations: Stepwise analysis showed that PALM maintained a significantly lower hallucination ratio as generation length increased, confirming that the Partial Attention mechanism keeps the model grounded in the source text.
  • Improved Generation Length: Standard LMs often suffer from "early stopping" as they lose track of the source. PALM’s average output lengths were much closer to human-gold standards and traditional ED models.

Conclusion

This paper reframes the decoder-only LM as a "regularized" version of the ED architecture. While these regularizations (like parameter sharing) reduce model size and can prevent overfitting, they also act as constraints that limit the model's ability to handle complex source-target relationships. By introducing PALM, the authors demonstrate that targeted structural changes can lead to more reliable and higher-performing generation without simply relying on increasing parameter counts. Future work will involve scaling PALM to the multi-billion parameter range to see how these structural advantages interact with massive pre-training.

Summary updated

Abstract

The sequence-to-sequence (seq2seq) task aims at generating the target sequence based on the given input source sequence. Traditionally, most of the seq2seq task is resolved by the Encoder-Decoder framework which requires an encoder to encode the source sequence and a decoder to generate the target text. Recently, a bunch of new approaches have emerged that apply decoder-only language models directly to the seq2seq task. Despite the significant advancements in applying language models to the seq2seq task, there is still a lack of thorough analysis on the effectiveness of the decoder-only language model architecture. This paper aims to address this gap by conducting a detailed comparison between the encoder-decoder architecture and the decoder-only language model framework through the analysis of a regularized encoder-decoder structure. This structure is designed to replicate all behaviors in the classical decoder-only language model but has an encoder and a decoder making it easier to be compared with the classical encoder-decoder structure. Based on the analysis, we unveil the attention degeneration problem in the language model, namely, as the generation step number grows, less and less attention is focused on the source sequence. To give a quantitative understanding of this problem, we conduct a theoretical sensitivity analysis of the attention output with respect to the source input. Grounded on our analysis, we propose a novel partial attention language model to solve the attention degeneration problem. Experimental results on machine translation, summarization, and data-to-text generation tasks support our analysis and demonstrate the effectiveness of our proposed model.
BibTeX
@misc{fu2023decoder,
  author = {Zihao Fu and Wai Lam and Qian Yu and Anthony Man-Cho So and Shengding Hu and Zhiyuan Liu and Nigel Collier},
  title = {{Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder}},
  year = {2023},
  eprint = {2304.04052},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.04052},
}
Figure