Abstract
The Transformer architecture is widely used in natural language processing. Despite its success, the design principle of the Transformer remains elusive. In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Ordinary Differential Equation (ODE) solver for a convection-diffusion equation in a multi-particle dynamic system. In particular, how words in a sentence are abstracted into contexts by passing through the layers of the Transformer can be interpreted as approximating multiple particles' movement in the space using the Lie-Trotter splitting scheme and the Euler's method. Given this ODE's perspective, the rich literature of numerical analysis can be brought to guide us in designing effective structures beyond the Transformer. As an example, we propose to replace the Lie-Trotter splitting scheme by the Strang-Marchuk splitting scheme, a scheme that is more commonly used and with much lower local truncation errors. This leads to a brand new architecture called the Macaron Net. Through extensive experiments, we show that the Macaron Net is superior to the Transformer on both supervised and unsupervised learning tasks.
Introduction
Despite the massive success of the Transformer in natural language processing, its design principles have remained largely empirical. Why do we use a self-attention layer followed by a feed-forward network (FFN)? Is this the optimal arrangement? This research seeks to move beyond "trial and error" architecture design by bridging the gap between deep learning and classical physics. By providing a theoretical framework that explains the Transformer's inner workings, we can derive more effective neural structures based on established numerical analysis.
The Multi-Particle Dynamic System Perspective
The core of this work is a novel perspective: word embeddings are particles moving through a high-dimensional semantic space. The transformation of word representations through layers can be modeled as a convection-diffusion equation in a Multi-Particle Dynamic System (MPDS):
- Diffusion (Interaction): The self-attention sub-layer represents the interaction between particles (words), allowing them to share information.
- Convection (Individual Movement): The position-wise FFN sub-layer represents the independent movement of each particle based on its own state.
- Time Steps: Each "layer" in the network corresponds to a discrete time step in a numerical ODE solver.
From ODEs to Macaron Net
To solve these complex equations, mathematicians use "splitting schemes." The standard Transformer can be interpreted as using the Lie-Trotter splitting scheme, which solves the diffusion and convection terms sequentially. This is a first-order approximation with relatively high truncation errors.
By contrast, we propose using the Strang-Marchuk splitting scheme. This suggests a symmetric arrangement: a half-step of convection, a full step of diffusion, and another half-step of convection. This mathematical shift results in the "Macaron" layer: [1/2 FFN] -> [Self-Attention] -> [1/2 FFN]. Each sub-layer retains residual connections. To keep the parameter count identical to a standard Transformer for fair comparison, the internal dimensionality of the FFNs is halved.
Experimental Results
We evaluated Macaron Net against standard Transformer and BERT baselines across both supervised and unsupervised tasks. For machine translation on WMT14 (En-De), Macaron Net "Big" achieved 30.2 BLEU, a significant improvement over the original Transformer Big's 28.4 BLEU.
On the GLUE benchmark, which tests general language understanding, Macaron Net outperformed BERT-base on almost every task. The overall GLUE score improved from 78.3 (BERT-base) to 79.7 (Macaron Net) using the same number of parameters and pre-training data.
Conclusion
The most significant insight of this work is that neural architecture design can be treated as a numerical analysis problem. By "cradling" the attention layer between two FFNs, the model achieves a higher-order approximation of the underlying system dynamics. The improvements in Macaron Net do not come from increasing model size, but from redistributing the same number of parameters into a more mathematically sound structure. Future work may explore even more sophisticated numerical methods, such as Runge-Kutta schemes, to inspire potentially more powerful architectures.
Summary updated