TL;DR
GTrans (Group-Transformer) is a novel architecture for Neural Machine Translation (NMT) that breaks the "top-layer-only" habit of standard Transformers. By grouping layers and fusing their representations at multiple levels, GTrans successfully scales to extremely deep models (up to 60 encoder layers) and consistently improves translation quality across bilingual and multilingual benchmarks.
What Problem Does This Paper Address?
Standard Transformer models are typically "top-heavy." They stack many layers but primarily rely on the final layer's representation for translation, assuming lower layers provide only redundant or trivial information. However, research suggests that these lower layers capture essential syntactic and low-level semantic features that are often lost by the time data reaches the top.
Furthermore, building deeper Transformers is notoriously difficult. Simply stacking more layers often leads to poor convergence and vanishing gradients, especially when using the high-performing "post-norm" residual units. While "pre-norm" units are easier to train, they generally underperform their post-norm counterparts. The challenge is: how do we effectively harvest features from all depths while making extremely deep models stable to train?
Methods
The authors introduce GTrans, which employs a "group-and-fuse" strategy for both the encoder and decoder.
- Grouping: Instead of treating every layer individually (dense fusion) or only using the last one, GTrans divides layers into groups (e.g., every 3 layers form a group).
- Encoder Fusion: For each encoder group, the last hidden state is selected. These group-level features are then fused into a single representation that the decoder attends to. This ensures the decoder can directly "see" information from various depths of the encoder.
- Decoder Fusion & Probability Aggregation: The decoder layers are also grouped. Crucially, instead of just one final prediction, each decoder group generates its own set of word probabilities. These probabilities are then combined using a weighted average to produce the final output. This allows low-level decoder features to contribute directly to the final word prediction.
Datasets
The model was rigorously tested on a wide variety of benchmarks:
- Bilingual: IWSLT-2014 (German→English), WMT-2014 (English→German), and LDC (Chinese→English).
- Multilingual: IWSLT-2017 (multiple European languages to/from English), OPUS-100 (a massive English-centric corpus covering 94 language pairs), and the WMT-2021 multilingual task.
Key Insights and Ideas
- Sparse is Better than Dense: For very deep models (e.g., 60 layers), fusing every layer (dense fusion) actually hurts performance and makes training unstable. Grouping layers (sparse fusion) acts as a regularizer and helps the model focus on the most useful hierarchical features.
- Direct Impact of Decoder Fusion: The authors found that decoder fusion has a more significant impact on translation quality than encoder fusion. Allowing intermediate decoder layers to "vote" on the final word probability is a powerful way to leverage multi-level features.
- Stabilizing Post-Norm: GTrans enables the successful training of very deep models using post-norm residual units, which are typically harder to optimize but yield better results than pre-norm units.
- Long Sentence Mastery: One of the most striking findings is that GTrans significantly outperforms standard Transformers on long sentences (>80 words), suggesting that multi-layer features are vital for maintaining context in complex translations.
Key Results
GTrans consistently outperformed vanilla Transformers and other state-of-the-art layer-fusion methods (like ReZero or DLCL):
- BLEU Gains: On WMT-2014 En→De, GTrans achieved a +0.79 BLEU improvement over the baseline without adding extra parameters.
- Extreme Scaling: The model scaled successfully to 60 encoder layers and 36 decoder layers, reaching 30.42 BLEU on WMT-14, showing that performance continues to climb with depth when using this fusion strategy.
- Efficiency: Despite the added complexity of fusion, GTrans maintains an inference speed very close to the standard Transformer (858 vs 884 words/sec) and uses the same number of parameters.
Limitations and Future Work
While GTrans is efficient, the authors note that increasing the number of decoder groups ($N$) can lead to higher training and inference times because each group must perform its own probability calculation.
Future research could explore more sophisticated, perhaps dynamic, grouping strategies rather than fixed group sizes. Additionally, investigating how GTrans performs in other sequence-to-sequence tasks like abstractive summarization or code generation could further prove its versatility.
Summary updated