TL;DR
While common wisdom suggests that fusing intermediate encoder layers helps Sequence-to-Sequence (Seq2Seq) models by capturing syntax, this research reveals that the encoder embedding layer is actually the most critical component. The authors introduce SurfaceFusion, a simple yet powerful method that directly connects source embeddings to the decoder’s output layer, achieving state-of-the-art results in machine translation and significant gains in summarization and error correction.
The Problem: The Source Representation Bottleneck
In standard Seq2Seq models like the Transformer, the decoder typically only "sees" the final representation produced by the encoder. "EncoderFusion" is a family of techniques designed to let the decoder access all encoder layers, which has been shown to improve performance.
However, the research community has been conflicted on why this works. Some argued that lower layers provide essential syntactic information, while others found that attending to intermediate layers didn't help at all. The authors identify a "source representation bottleneck": the final encoder layer is so focused on abstract semantics that it loses the "surface-level" (raw word) information necessary for precise token prediction.
Methods: Diagnosis and Solution
The authors tackle this problem in two stages: diagnosis and solution.
1. Fine-grained Layer Attention (FGLA)
To investigate which layers actually matter, they developed a diagnostic tool that assigns individual attention weights to every dimension of every encoder layer. This allowed them to visualize exactly where the decoder was looking during generation.
2. SurfaceFusion
Based on their findings that the embedding layer is the "MVP" of fusion, they proposed a simplified architecture. Instead of complex multi-layer fusion, SurfaceFusion uses a multi-head attention module to extract a "surface representation" directly from the encoder's embedding layer. This representation is then fused with the decoder's final output using one of two methods:
- Hard Fusion: A weighted interpolation of the probabilities.
- Soft Fusion: Combining the values at the logit level before the final softmax.
Key Insights and Ideas
The Embedding Layer is the Secret Sauce: Visualization of the FGLA weights showed that the uppermost decoder layers—the ones responsible for picking the actual word—consistently pay the most attention to the encoder’s raw embedding layer, largely ignoring the intermediate layers.
Masking Reveals Function: The authors performed "ablation by masking" to understand the distinct roles of different layers. They found that masking the encoder's top layer leads to omissions (the model forgets to translate parts of the sentence). Conversely, masking the embedding layer leads to hallucinations (the model produces fluent but completely unfaithful text).
Expressivity Matters: Using Singular Value Decomposition (SVD), the authors proved that the dimensions in the embedding layer the decoder attends to are more "expressive" (mathematically diverse) than those it ignores.
Shortening the Path: By connecting the source embeddings directly to the output, SurfaceFusion forces the model to learn a closer relationship between source and target words in the vector space, effectively "aligning" the two languages better.
Experimental Results
The researchers evaluated their approach across three distinct NLP tasks to ensure universality: Machine Translation (WMT16 Ro-En, WMT14 En-De, WMT14 En-Fr), Text Summarization (CNN/Daily Mail), and Grammatical Error Correction (CoNLL14).
SurfaceFusion consistently outperformed both vanilla Transformers and more complex "all-layer" fusion methods:
- State-of-the-Art (SOTA): Achieved SOTA results on WMT16 Ro-En (35.1 BLEU) and WMT14 En-Fr (43.9 BLEU).
- Broad Success: In Grammatical Error Correction, it improved the $F_{0.5}$ score from 54.3 to 56.6. In summarization, it saw a boost across all ROUGE metrics.
- Efficiency: Despite the performance gains, SurfaceFusion only introduces about a 10% increase in inference latency, making it much more practical than methods that fuse every single encoder layer.
Limitations and Future Work
While SurfaceFusion is highly effective, the authors note a few areas for future exploration:
- Inference Overhead: While a 10% latency increase is low compared to other fusion methods, it is still a cost that production environments must weigh against accuracy gains.
- Task Expansion: The authors suggest validating these findings on non-text Seq2Seq tasks, such as speech recognition or dialogue systems.
- Architectural Alternatives: Future work could explore if there are even more efficient ways to bypass the "source representation bottleneck" without requiring an additional attention module.
Summary updated