Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning

Xuebo Liu, Longyue Wang, Derek F. Wong, Liang Ding, Lidia S. Chao, Zhaopeng Tu

TL;DR

While common wisdom suggests that fusing intermediate encoder layers helps Sequence-to-Sequence (Seq2Seq) models by capturing syntax, this research reveals that the encoder embedding layer is actually the most critical component. The authors introduce SurfaceFusion, a simple yet powerful method that directly connects source embeddings to the decoder’s output layer, achieving state-of-the-art results in machine translation and significant gains in summarization and error correction.

The Problem: The Source Representation Bottleneck

In standard Seq2Seq models like the Transformer, the decoder typically only "sees" the final representation produced by the encoder. "EncoderFusion" is a family of techniques designed to let the decoder access all encoder layers, which has been shown to improve performance.

However, the research community has been conflicted on why this works. Some argued that lower layers provide essential syntactic information, while others found that attending to intermediate layers didn't help at all. The authors identify a "source representation bottleneck": the final encoder layer is so focused on abstract semantics that it loses the "surface-level" (raw word) information necessary for precise token prediction.

Methods: Diagnosis and Solution

The authors tackle this problem in two stages: diagnosis and solution.

1. Fine-grained Layer Attention (FGLA)

To investigate which layers actually matter, they developed a diagnostic tool that assigns individual attention weights to every dimension of every encoder layer. This allowed them to visualize exactly where the decoder was looking during generation.

2. SurfaceFusion

Based on their findings that the embedding layer is the "MVP" of fusion, they proposed a simplified architecture. Instead of complex multi-layer fusion, SurfaceFusion uses a multi-head attention module to extract a "surface representation" directly from the encoder's embedding layer. This representation is then fused with the decoder's final output using one of two methods:

  • Hard Fusion: A weighted interpolation of the probabilities.
  • Soft Fusion: Combining the values at the logit level before the final softmax.

Key Insights and Ideas

The Embedding Layer is the Secret Sauce: Visualization of the FGLA weights showed that the uppermost decoder layers—the ones responsible for picking the actual word—consistently pay the most attention to the encoder’s raw embedding layer, largely ignoring the intermediate layers.

Attention weights across different tasks show top decoder layers heavily attend to the encoder embedding layer.

Masking Reveals Function: The authors performed "ablation by masking" to understand the distinct roles of different layers. They found that masking the encoder's top layer leads to omissions (the model forgets to translate parts of the sentence). Conversely, masking the embedding layer leads to hallucinations (the model produces fluent but completely unfaithful text).

Relative change in model performance when masking different encoder layers. Masking the embedding layer (red) or the top layer (green) causes the most significant performance drops.

Expressivity Matters: Using Singular Value Decomposition (SVD), the authors proved that the dimensions in the embedding layer the decoder attends to are more "expressive" (mathematically diverse) than those it ignores.

Shortening the Path: By connecting the source embeddings directly to the output, SurfaceFusion forces the model to learn a closer relationship between source and target words in the vector space, effectively "aligning" the two languages better.

Experimental Results

The researchers evaluated their approach across three distinct NLP tasks to ensure universality: Machine Translation (WMT16 Ro-En, WMT14 En-De, WMT14 En-Fr), Text Summarization (CNN/Daily Mail), and Grammatical Error Correction (CoNLL14).

SurfaceFusion consistently outperformed both vanilla Transformers and more complex "all-layer" fusion methods:

  • State-of-the-Art (SOTA): Achieved SOTA results on WMT16 Ro-En (35.1 BLEU) and WMT14 En-Fr (43.9 BLEU).
  • Broad Success: In Grammatical Error Correction, it improved the $F_{0.5}$ score from 54.3 to 56.6. In summarization, it saw a boost across all ROUGE metrics.
  • Efficiency: Despite the performance gains, SurfaceFusion only introduces about a 10% increase in inference latency, making it much more practical than methods that fuse every single encoder layer.

Limitations and Future Work

While SurfaceFusion is highly effective, the authors note a few areas for future exploration:

  • Inference Overhead: While a 10% latency increase is low compared to other fusion methods, it is still a cost that production environments must weigh against accuracy gains.
  • Task Expansion: The authors suggest validating these findings on non-text Seq2Seq tasks, such as speech recognition or dialogue systems.
  • Architectural Alternatives: Future work could explore if there are even more efficient ways to bypass the "source representation bottleneck" without requiring an additional attention module.

Summary updated

Abstract

Encoder layer fusion (EncoderFusion) is a technique to fuse all the encoder layers (instead of the uppermost layer) for sequence-to-sequence (Seq2Seq) models, which has proven effective on various NLP tasks. However, it is still not entirely clear why and when EncoderFusion should work. In this paper, our main contribution is to take a step further in understanding EncoderFusion. Many of previous studies believe that the success of EncoderFusion comes from exploiting surface and syntactic information embedded in lower encoder layers. Unlike them, we find that the encoder embedding layer is more important than other intermediate encoder layers. In addition, the uppermost decoder layer consistently pays more attention to the encoder embedding layer across NLP tasks. Based on this observation, we propose a simple fusion method, SurfaceFusion, by fusing only the encoder embedding layer for the softmax layer. Experimental results show that SurfaceFusion outperforms EncoderFusion on several NLP benchmarks, including machine translation, text summarization, and grammatical error correction. It obtains the state-of-the-art performance on WMT16 Romanian-English and WMT14 English-French translation tasks. Extensive analyses reveal that SurfaceFusion learns more expressive bilingual word embeddings by building a closer relationship between relevant source and target embeddings. Source code is freely available at https://github.com/SunbowLiu/SurfaceFusion.
BibTeX
@inproceedings{liu2021understanding,
  author = {Xuebo Liu and Longyue Wang and Derek F. Wong and Liang Ding and Lidia S. Chao and Zhaopeng Tu},
  title = {{Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=n1HD8M6WGn},
}
Figure