Multi-Path Transformer is Better: A Case Study on Neural Machine Translation

Ye Lin, Shuhan Zhou, Yanyang Li, Anxiang Ma, Tong Xiao, Jingbo Zhu

Abstract: For years the model performance in machine learning obeyed a power-law relationship with the model size. For the consideration of parameter efficiency, recent studies focus on increasing model depth rather than width to achieve better performance. In this paper, we study how model width affects the Transformer model through a parameter-efficient multi-path structure. To better fuse features extracted from different paths, we add three additional operations to each sublayer: a normalization at the end of each path, a cheap operation to produce more features, and a learnable weighted mechanism to fuse all features flexibly. Extensive experiments on 12 WMT machine translation tasks show that, with the same number of parameters, the shallower multi-path model can achieve similar or even better performance than the deeper model. It reveals that we should pay more attention to the multi-path structure, and there should be a balance between the model depth and width to train a better large-scale Transformer.

Introduction

In the quest for better Neural Machine Translation (NMT), researchers typically follow two paths: making models "wider" by increasing matrix dimensions (like moving from Transformer-base to Transformer-big) or making them "deeper" by stacking more layers.

Both approaches have significant drawbacks. Increasing matrix dimensions causes computational costs to grow quadratically ($O(d^2)$). Conversely, increasing depth provides more non-linearity but eventually hits diminishing returns and makes training unstable. This paper investigates a third way: Multi-path structures. Instead of one massive layer or dozens of thin layers, the authors propose using several parallel "paths" within each layer, optimizing a concept previously successful in Computer Vision (e.g., ResNeXt) for the Transformer architecture.

Methods

The core of the proposal is a redesigned multi-path Transformer sub-layer. Instead of a single Multi-Head Attention (MHA) or Feed-Forward Network (FFN), the architecture utilizes multiple parallel versions integrated with three key innovations:

  1. PathNorm: A normalization step at the end of each individual path. This ensures the magnitude of output distributions remains consistent, facilitating the fusion of features from different paths.
  2. Learnable Weighted Mechanism: Rather than simple averaging, the model learns weights ($\alpha$) for each path and a weight ($\beta$) for the residual connection. This allows the model to dynamically prioritize specific paths during training.
  3. Cheap Operations ("Selection then Combination"): To increase feature count without exploding parameters, the authors use a "selection then combination" strategy. They take existing "raw" features from $n$ paths, create subsets, and average them to produce "new" features, providing richer representations at near-zero additional cost.
Architecture of the Multi-Path Transformer sub-layer showing parallel MHA/FFN paths and learnable fusion.

Key Insights and Results

The researchers evaluated their approach on 12 WMT machine translation tasks, including high-competition benchmarks like WMT14 En↔De and En↔Fr.

Performance and Efficiency

The most striking finding is that a shallower multi-path model can outperform a model twice its depth. For example, a multi-path model with 24 layers outperformed a 48-layer "deep" model of the same parameter size. On WMT14 En→De, the multi-path model achieved 29.68 BLEU, which is 2.68 points higher than the Transformer-base baseline.

BLEU score performance vs. parameter count, demonstrating that the Multi-Path architecture consistently outperforms deep and wide alternatives.

Scaling and Stability

  • Linear Scaling: Unlike "wide" models where costs grow quadratically with dimension, multi-path models scale linearly with the number of paths, offering a more efficient way to widen a network.
  • Diversity: Analysis of learnable weights suggests that the model develops higher feature diversity in the middle layers, indicating that multi-path structures are particularly adept at capturing complex internal representations.
  • Training Stability: The combination of PathNorm and learnable weights helped prevent "loss exploding" issues often seen in very deep models during mixed-precision training.

Limitations and Future Work

While theoretically efficient, current deep learning libraries are optimized for large matrix multiplications rather than many small parallel ones, meaning practical training times currently lag behind theoretical ideals. Additionally, while this study focused on models up to ~250M parameters, further research is needed to see how these benefits translate to "Very Large-Scale" models (billions of parameters) where memory bandwidth becomes the primary bottleneck.

Summary updated

Abstract

For years the model performance in machine learning obeyed a power-law relationship with the model size. For the consideration of parameter efficiency, recent studies focus on increasing model depth rather than width to achieve better performance. In this paper, we study how model width affects the Transformer model through a parameter-efficient multi-path structure. To better fuse features extracted from different paths, we add three additional operations to each sublayer: a normalization at the end of each path, a cheap operation to produce more features, and a learnable weighted mechanism to fuse all features flexibly. Extensive experiments on 12 WMT machine translation tasks show that, with the same number of parameters, the shallower multi-path model can achieve similar or even better performance than the deeper model. It reveals that we should pay more attention to the multi-path structure, and there should be a balance between the model depth and width to train a better large-scale Transformer.
BibTeX
@inproceedings{lin2022multi,
  author = {Ye Lin and Shuhan Zhou and Yanyang Li and Anxiang Ma and Tong Xiao and Jingbo Zhu},
  title = {{Multi-Path Transformer is Better: A Case Study on Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=eWPU_QiRWXi},
}
Figure