Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View

Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, Tie-Yan LiuICLR 2019 · OpenReview

Abstract

The Transformer architecture is widely used in natural language processing. Despite its success, the design principle of the Transformer remains elusive. In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Ordinary Differential Equation (ODE) solver for a convection-diffusion equation in a multi-particle dynamic system. In particular, how words in a sentence are abstracted into contexts by passing through the layers of the Transformer can be interpreted as approximating multiple particles' movement in the space using the Lie-Trotter splitting scheme and the Euler's method. Given this ODE's perspective, the rich literature of numerical analysis can be brought to guide us in designing effective structures beyond the Transformer. As an example, we propose to replace the Lie-Trotter splitting scheme by the Strang-Marchuk splitting scheme, a scheme that is more commonly used and with much lower local truncation errors. This leads to a brand new architecture called the Macaron Net. Through extensive experiments, we show that the Macaron Net is superior to the Transformer on both supervised and unsupervised learning tasks.

Introduction

Despite the massive success of the Transformer in natural language processing, its design principles have remained largely empirical. Why do we use a self-attention layer followed by a feed-forward network (FFN)? Is this the optimal arrangement? This research seeks to move beyond "trial and error" architecture design by bridging the gap between deep learning and classical physics. By providing a theoretical framework that explains the Transformer's inner workings, we can derive more effective neural structures based on established numerical analysis.

The Multi-Particle Dynamic System Perspective

The core of this work is a novel perspective: word embeddings are particles moving through a high-dimensional semantic space. The transformation of word representations through layers can be modeled as a convection-diffusion equation in a Multi-Particle Dynamic System (MPDS):

  • Diffusion (Interaction): The self-attention sub-layer represents the interaction between particles (words), allowing them to share information.
  • Convection (Individual Movement): The position-wise FFN sub-layer represents the independent movement of each particle based on its own state.
  • Time Steps: Each "layer" in the network corresponds to a discrete time step in a numerical ODE solver.

From ODEs to Macaron Net

To solve these complex equations, mathematicians use "splitting schemes." The standard Transformer can be interpreted as using the Lie-Trotter splitting scheme, which solves the diffusion and convection terms sequentially. This is a first-order approximation with relatively high truncation errors.

By contrast, we propose using the Strang-Marchuk splitting scheme. This suggests a symmetric arrangement: a half-step of convection, a full step of diffusion, and another half-step of convection. This mathematical shift results in the "Macaron" layer: [1/2 FFN] -> [Self-Attention] -> [1/2 FFN]. Each sub-layer retains residual connections. To keep the parameter count identical to a standard Transformer for fair comparison, the internal dimensionality of the FFNs is halved.

Comparison between the original Transformer and the proposed Macaron Net architecture.

Experimental Results

We evaluated Macaron Net against standard Transformer and BERT baselines across both supervised and unsupervised tasks. For machine translation on WMT14 (En-De), Macaron Net "Big" achieved 30.2 BLEU, a significant improvement over the original Transformer Big's 28.4 BLEU.

On the GLUE benchmark, which tests general language understanding, Macaron Net outperformed BERT-base on almost every task. The overall GLUE score improved from 78.3 (BERT-base) to 79.7 (Macaron Net) using the same number of parameters and pre-training data.

Performance comparison on the GLUE benchmark.

Conclusion

The most significant insight of this work is that neural architecture design can be treated as a numerical analysis problem. By "cradling" the attention layer between two FFNs, the model achieves a higher-order approximation of the underlying system dynamics. The improvements in Macaron Net do not come from increasing model size, but from redistributing the same number of parameters into a more mathematically sound structure. Future work may explore even more sophisticated numerical methods, such as Runge-Kutta schemes, to inspire potentially more powerful architectures.

Summary updated

Abstract

The Transformer architecture is widely used in natural language processing. Despite its success, the design principle of the Transformer remains elusive. In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Ordinary Differential Equation (ODE) solver for a convection-diffusion equation in a multi-particle dynamic system. In particular, how words in a sentence are abstracted into contexts by passing through the layers of the Transformer can be interpreted as approximating multiple particles' movement in the space using the Lie-Trotter splitting scheme and the Euler's method. Given this ODE's perspective, the rich literature of numerical analysis can be brought to guide us in designing effective structures beyond the Transformer. As an example, we propose to replace the Lie-Trotter splitting scheme by the Strang-Marchuk splitting scheme, a scheme that is more commonly used and with much lower local truncation errors. The Strang-Marchuk splitting scheme suggests that the self-attention and position-wise feed-forward network (FFN) sub-layers should not be treated equally. Instead, in each layer, two position-wise FFN sub-layers should be used, and the self-attention sub-layer is placed in between. This leads to a brand new architecture. Such an FFN-attention-FFN layer is "Macaron-like", and thus we call the network with this new architecture the Macaron Net. Through extensive experiments, we show that the Macaron Net is superior to the Transformer on both supervised and unsupervised learning tasks. The reproducible codes and pretrained models can be found at https://github.com/zhuohan123/macaron-net
BibTeX
@inproceedings{lu2019understanding,
  author = {Yiping Lu and Zhuohan Li and Di He and Zhiqing Sun and Bin Dong and Tao Qin and Liwei Wang and Tie-Yan Liu},
  title = {{Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=6Sjk90aVNK},
}
Figure