GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation

Jian Yang, Yuwei Yin, Liqun Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Furu Wei, Zhoujun Li

TL;DR

GTrans (Group-Transformer) is a novel architecture for Neural Machine Translation (NMT) that breaks the "top-layer-only" habit of standard Transformers. By grouping layers and fusing their representations at multiple levels, GTrans successfully scales to extremely deep models (up to 60 encoder layers) and consistently improves translation quality across bilingual and multilingual benchmarks.

What Problem Does This Paper Address?

Standard Transformer models are typically "top-heavy." They stack many layers but primarily rely on the final layer's representation for translation, assuming lower layers provide only redundant or trivial information. However, research suggests that these lower layers capture essential syntactic and low-level semantic features that are often lost by the time data reaches the top.

Furthermore, building deeper Transformers is notoriously difficult. Simply stacking more layers often leads to poor convergence and vanishing gradients, especially when using the high-performing "post-norm" residual units. While "pre-norm" units are easier to train, they generally underperform their post-norm counterparts. The challenge is: how do we effectively harvest features from all depths while making extremely deep models stable to train?

Methods

The authors introduce GTrans, which employs a "group-and-fuse" strategy for both the encoder and decoder.

  1. Grouping: Instead of treating every layer individually (dense fusion) or only using the last one, GTrans divides layers into groups (e.g., every 3 layers form a group).
  2. Encoder Fusion: For each encoder group, the last hidden state is selected. These group-level features are then fused into a single representation that the decoder attends to. This ensures the decoder can directly "see" information from various depths of the encoder.
  3. Decoder Fusion & Probability Aggregation: The decoder layers are also grouped. Crucially, instead of just one final prediction, each decoder group generates its own set of word probabilities. These probabilities are then combined using a weighted average to produce the final output. This allows low-level decoder features to contribute directly to the final word prediction.
Fig. 1. Comparison between (a) the single-layer feature and (b) multi-layer feature fusion. (a) is the network only using representation of the last layer, and (b) is the network with the multi-layer feature incorporation, where all layers are divided into different groups.
Fig. 2. Overview of our proposed model, where the encoder has 6 layers with 3 groups and decoder has 6 layers with 2 groups. Given the source input x, our model splits a sequence of stacked encoder layers into 3 groups and selects the last representation of each group for fusion. Similarly, the decoder layers are divided into 2 groups, where each group has the corresponding fused representation. All fused features of each decoder group are used to separately generate probabilities, which are combined to predict the target word yi.

Datasets

The model was rigorously tested on a wide variety of benchmarks:

  • Bilingual: IWSLT-2014 (German→English), WMT-2014 (English→German), and LDC (Chinese→English).
  • Multilingual: IWSLT-2017 (multiple European languages to/from English), OPUS-100 (a massive English-centric corpus covering 94 language pairs), and the WMT-2021 multilingual task.

Key Insights and Ideas

  • Sparse is Better than Dense: For very deep models (e.g., 60 layers), fusing every layer (dense fusion) actually hurts performance and makes training unstable. Grouping layers (sparse fusion) acts as a regularizer and helps the model focus on the most useful hierarchical features.
  • Direct Impact of Decoder Fusion: The authors found that decoder fusion has a more significant impact on translation quality than encoder fusion. Allowing intermediate decoder layers to "vote" on the final word probability is a powerful way to leverage multi-level features.
  • Stabilizing Post-Norm: GTrans enables the successful training of very deep models using post-norm residual units, which are typically harder to optimize but yield better results than pre-norm units.
  • Long Sentence Mastery: One of the most striking findings is that GTrans significantly outperforms standard Transformers on long sentences (>80 words), suggesting that multi-layer features are vital for maintaining context in complex translations.

Key Results

GTrans consistently outperformed vanilla Transformers and other state-of-the-art layer-fusion methods (like ReZero or DLCL):

  • BLEU Gains: On WMT-2014 En→De, GTrans achieved a +0.79 BLEU improvement over the baseline without adding extra parameters.
  • Extreme Scaling: The model scaled successfully to 60 encoder layers and 36 decoder layers, reaching 30.42 BLEU on WMT-14, showing that performance continues to climb with depth when using this fusion strategy.
  • Efficiency: Despite the added complexity of fusion, GTrans maintains an inference speed very close to the standard Transformer (858 vs 884 words/sec) and uses the same number of parameters.
Fig. 3. Effect of the number of the encoder layers with the fixed 12 decoder layers (a) and the number of the decoder layers with the fixed 12 encoder layers (b) on the IWSLT-2014 De→En translation task.

Limitations and Future Work

While GTrans is efficient, the authors note that increasing the number of decoder groups ($N$) can lead to higher training and inference times because each group must perform its own probability calculation.

Future research could explore more sophisticated, perhaps dynamic, grouping strategies rather than fixed group sizes. Additionally, investigating how GTrans performs in other sequence-to-sequence tasks like abstractive summarization or code generation could further prove its versatility.

Summary updated

Abstract

Transformer structure, stacked by a sequence of encoder and decoder network layers, achieves significant development in neural machine translation. However, vanilla Transformer mainly exploits the top-layer representation, assuming the lower layers provide trivial or redundant information and thus ignoring the bottom-layer feature that is potentially valuable. In this work, we propose the Group-Transformer model (GTrans) that flexibly divides multi-layer representations of both encoder and decoder into different groups and then fuses these group features to generate target words. To corroborate the effectiveness of the proposed method, extensive experiments and analytic experiments are conducted on three bilingual translation benchmarks and two multilingual translation tasks, including the IWLST-14, IWLST-17, LDC, WMT-14 and OPUS-100 benchmark. Experimental and analytical results demonstrate that our model outperforms its Transformer counterparts by a consistent gain. Furthermore, it can be successfully scaled up to 60 encoder layers and 36 decoder layers.
BibTeX
@article{yang2023gtrans,
  author = {Jian Yang and Yuwei Yin and Liqun Yang and Shuming Ma and Haoyang Huang and Dongdong Zhang and Furu Wei and Zhoujun Li},
  title = {{GTrans: Grouping and Fusing Transformer Layers for Neural Machine Translation}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2023},
  url = {https://openreview.net/forum?id=XkBw6nlg5Q},
}
Figure