Abstract
While decoder-only language models (LMs) like GPT have become the standard for large-scale pre-training, they often underperform traditional encoder-decoder (ED) architectures on sequence-to-sequence (seq2seq) tasks. This paper investigates this performance gap by introducing the Regularized Encoder-Decoder (RED) framework, which interprets LMs as ED models subject to specific structural regularizations. The authors identify the Attention Degeneration Problem (ADP)—a mathematical decay in sensitivity to source inputs as generation length increases—and propose the Partial Attention Language Model (PALM) to mitigate these issues. Experimental results across translation and summarization benchmarks demonstrate that PALM achieves superior performance and grounding compared to standard LMs.
Introduction
In recent years, the Natural Language Processing landscape has shifted from specialized Encoder-Decoder (ED) models (e.g., T5, BART) toward unified Decoder-Only Language Models (LM) (e.g., GPT-4, Llama). While LMs are easier to scale and pre-train, they frequently struggle with seq2seq tasks like machine translation and summarization, often suffering from hallucinations or premature stopping. This research seeks to understand the structural reasons behind this performance gap by asking: What is the mathematical difference between an LM and an ED model, and how does that difference affect performance?
The RED Framework and Attention Degeneration
To bridge the gap between these architectures, the authors propose the Regularized Encoder-Decoder (RED) framework. RED is a theoretical construct that configures an ED model to behave exactly like a decoder-only LM. This allows researchers to isolate specific "regularizations" inherent in LMs, such as parameter sharing between the encoder and decoder components and the use of consecutive positional encoding across the entire sequence.
Through sensitivity analysis using Jacobian matrices, the authors mathematically prove the existence of the Attention Degeneration Problem (ADP). They demonstrate that in a standard LM, the output's sensitivity to the source input decreases as the generation step increases. Essentially, as the model generates more tokens, the "attention budget" for the original source text is diluted by the growing target sequence, causing the model to "forget" the prompt.
PALM: The Partial Attention Language Model
To address ADP, the authors introduce the Partial Attention Language Model (PALM). PALM incorporates three key structural modifications:
- Partial Attention (PA): A dedicated attention component that focuses exclusively on the source sequence, ensuring the source signal remains strong regardless of output length.
- Bidirectional Source Attention: Unlike the causal masking in standard LMs, PALM allows tokens in the source sequence to attend to each other bidirectionally.
- Positional Restart: PALM restarts positional encodings for the target sequence, helping the model better distinguish between the source and target domains.
Experimental Results
The researchers evaluated PALM across standard benchmarks including IWSLT’14, WMT’16, and XSUM. Key findings include:
- Efficiency and Accuracy: PALM consistently outperformed standard LMs. Notably, a PALM model with 35.6M parameters often outperformed traditional ED models with 47.1M parameters on translation tasks.
- Reduced Hallucinations: Stepwise analysis showed that PALM maintained a significantly lower hallucination ratio as generation length increased, confirming that the Partial Attention mechanism keeps the model grounded in the source text.
- Improved Generation Length: Standard LMs often suffer from "early stopping" as they lose track of the source. PALM’s average output lengths were much closer to human-gold standards and traditional ED models.
Conclusion
This paper reframes the decoder-only LM as a "regularized" version of the ED architecture. While these regularizations (like parameter sharing) reduce model size and can prevent overfitting, they also act as constraints that limit the model's ability to handle complex source-target relationships. By introducing PALM, the authors demonstrate that targeted structural changes can lead to more reliable and higher-performing generation without simply relying on increasing parameter counts. Future work will involve scaling PALM to the multi-billion parameter range to see how these structural advantages interact with massive pre-training.
Summary updated