Multi-Path Transformer is Better: A Case Study on Neural Machine Translation

Ye Lin, Shuhan Zhou, Yanyang Li, Anxiang Ma, Tong Xiao, Jingbo Zhu

Abstract

Researchers from Northeastern University and NiuTrans propose a sublayer-level multi-path Transformer architecture that challenges the prevailing "deeper is better" paradigm in neural machine translation (NMT). By utilizing parallel paths with specialized normalization (PathNorm) and learnable fusion weights, the proposed model achieves state-of-the-art results across multiple benchmarks. Their findings indicate that a shallower, multi-path "wide" model can outperform significantly deeper "narrow" models of the same parameter count while maintaining superior training stability.

Introduction

In the pursuit of higher performance, the field of deep learning has largely focused on increasing model depth. While adding layers provides the non-linearity necessary for complex tasks, it often encounters diminishing returns and training instabilities such as vanishing or exploding gradients. An alternative approach—increasing model width—is frequently overlooked because traditional widening (increasing internal matrix dimensions) causes computational costs to grow quadratically.

This paper explores an efficient alternative: multi-path structures. Instead of a single sequential processing stream, the model employs several parallel "lanes" per layer. This approach allows for increased model capacity with only linear growth in parameters and computation, provided that the information from these paths can be effectively coordinated and fused.

Methodology

The authors introduce a redesigned multi-path Transformer that applies parallel structures to both the Multi-Head Attention (MHA) and Feed-Forward Network (FFN) sublayers. The architecture incorporates three core innovations to ensure stability and performance:

  1. PathNorm: A normalization layer is added to the end of every individual path. This ensures that output distributions from different paths have similar magnitudes, facilitating easier fusion.
  2. Learnable Weighted Mechanism: Rather than simple averaging, the model learns two sets of weights: $\alpha$ (to weight each individual path) and $\beta$ (to weight the residual connection). This allows the network to dynamically prioritize specific information paths during training.
  3. Selection then Combination: To further enrich feature representation without adding parameters, the model creates "new" features by averaging different subsets of the existing "raw" paths.
The proposed multi-path Transformer architecture featuring parallel MHA and FFN sublayers with PathNorm and learnable weights.

Experimental Results

The researchers evaluated their approach on 12 WMT benchmarks, including high-resource pairs like English↔German (En↔De) and mid-resource pairs like English↔Latvian (En↔Lv).

Performance and Efficiency

The multi-path model consistently outperformed both standard deep models and traditional "wide" models. On the WMT14 En→De task, the multi-path architecture achieved a BLEU score 2.65 points higher than the standard Transformer-base. Notably, a 24-layer multi-path model achieved a BLEU score of 29.68, surpassing a much deeper 48-layer standard model of equivalent parameter size.

BLEU score performance vs. parameter count, comparing the proposed Multi-Path approach against Deep, Wide, and standard Multi-Path baselines.

Training Stability and Path Diversity

The combination of PathNorm and learnable weights effectively stabilized training for deep configurations. In scenarios where standard deep models failed to converge (e.g., WMT14 Fr→En), the multi-path model trained successfully. Analysis of the learnable weights ($\alpha$) revealed that path diversity is highest in the middle layers of the network, suggesting that these layers are critical for complex feature fusion.

Discussion and Conclusion

This study demonstrates that model width, achieved through multi-path structures, is as critical as depth for NMT performance. The multi-path approach offers a more efficient scaling path than traditional widening or deepening.

However, some limitations remain. Current deep learning libraries are not yet fully optimized for parallel path execution, leading to a gap between theoretical and practical training speeds. Additionally, while the approach was validated up to ~200M parameters, its effectiveness at the multi-billion parameter scale remains a subject for future research. Nonetheless, the Multi-Path Transformer represents a significant step toward more robust and efficient neural architecture design.

Summary updated

Abstract

For years the model performance in machine learning obeyed a power-law relationship with the model size. For the consideration of parameter efficiency, recent studies focus on increasing model depth rather than width to achieve better performance. In this paper, we study how model width affects the Transformer model through a parameter-efficient multi-path structure. To better fuse features extracted from different paths, we add three additional operations to each sublayer: a normalization at the end of each path, a cheap operation to produce more features, and a learnable weighted mechanism to fuse all features flexibly. Extensive experiments on 12 WMT machine translation tasks show that, with the same number of parameters, the shallower multi-path model can achieve similar or even better performance than the deeper model. It reveals that we should pay more attention to the multi-path structure, and there should be a balance between the model depth and width to train a better large-scale Transformer.
BibTeX
@inproceedings{lin2022multi,
  author = {Ye Lin and Shuhan Zhou and Yanyang Li and Anxiang Ma and Tong Xiao and Jingbo Zhu},
  title = {{Multi-Path Transformer is Better: A Case Study on Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=eWPU_QiRWXi},
}
Figure