Abstract
Researchers from Northeastern University and NiuTrans propose a sublayer-level multi-path Transformer architecture that challenges the prevailing "deeper is better" paradigm in neural machine translation (NMT). By utilizing parallel paths with specialized normalization (PathNorm) and learnable fusion weights, the proposed model achieves state-of-the-art results across multiple benchmarks. Their findings indicate that a shallower, multi-path "wide" model can outperform significantly deeper "narrow" models of the same parameter count while maintaining superior training stability.
Introduction
In the pursuit of higher performance, the field of deep learning has largely focused on increasing model depth. While adding layers provides the non-linearity necessary for complex tasks, it often encounters diminishing returns and training instabilities such as vanishing or exploding gradients. An alternative approach—increasing model width—is frequently overlooked because traditional widening (increasing internal matrix dimensions) causes computational costs to grow quadratically.
This paper explores an efficient alternative: multi-path structures. Instead of a single sequential processing stream, the model employs several parallel "lanes" per layer. This approach allows for increased model capacity with only linear growth in parameters and computation, provided that the information from these paths can be effectively coordinated and fused.
Methodology
The authors introduce a redesigned multi-path Transformer that applies parallel structures to both the Multi-Head Attention (MHA) and Feed-Forward Network (FFN) sublayers. The architecture incorporates three core innovations to ensure stability and performance:
- PathNorm: A normalization layer is added to the end of every individual path. This ensures that output distributions from different paths have similar magnitudes, facilitating easier fusion.
- Learnable Weighted Mechanism: Rather than simple averaging, the model learns two sets of weights: $\alpha$ (to weight each individual path) and $\beta$ (to weight the residual connection). This allows the network to dynamically prioritize specific information paths during training.
- Selection then Combination: To further enrich feature representation without adding parameters, the model creates "new" features by averaging different subsets of the existing "raw" paths.
Experimental Results
The researchers evaluated their approach on 12 WMT benchmarks, including high-resource pairs like English↔German (En↔De) and mid-resource pairs like English↔Latvian (En↔Lv).
Performance and Efficiency
The multi-path model consistently outperformed both standard deep models and traditional "wide" models. On the WMT14 En→De task, the multi-path architecture achieved a BLEU score 2.65 points higher than the standard Transformer-base. Notably, a 24-layer multi-path model achieved a BLEU score of 29.68, surpassing a much deeper 48-layer standard model of equivalent parameter size.
Training Stability and Path Diversity
The combination of PathNorm and learnable weights effectively stabilized training for deep configurations. In scenarios where standard deep models failed to converge (e.g., WMT14 Fr→En), the multi-path model trained successfully. Analysis of the learnable weights ($\alpha$) revealed that path diversity is highest in the middle layers of the network, suggesting that these layers are critical for complex feature fusion.
Discussion and Conclusion
This study demonstrates that model width, achieved through multi-path structures, is as critical as depth for NMT performance. The multi-path approach offers a more efficient scaling path than traditional widening or deepening.
However, some limitations remain. Current deep learning libraries are not yet fully optimized for parallel path execution, leading to a gap between theoretical and practical training speeds. Additionally, while the approach was validated up to ~200M parameters, its effectiveness at the multi-billion parameter scale remains a subject for future research. Nonetheless, the Multi-Path Transformer represents a significant step toward more robust and efficient neural architecture design.
Summary updated