Abstract
Transformer models have become foundational in AI, yet the theoretical understanding of their training dynamics remains incomplete. This paper investigates the convergence of gradient descent on a structurally complete single-layer Transformer. We prove that under appropriate initialization, gradient descent achieves a linear convergence rate. Crucially, we identify that residual connections are vital for maintaining the rank of the attention output, thereby preventing ill-conditioning and ensuring optimization stability.
Introduction
Despite the empirical dominance of Transformers, mathematical analysis often simplifies the architecture by isolating components like self-attention or feedforward networks (FFNs). This study bridges that gap by analyzing the interplay between self-attention, FFNs, and residual connections. The central question is how these components collectively influence the optimization landscape to reach a global minimum.
Theoretical Framework
The researchers analyzed a complete Transformer block consisting of softmax self-attention, a two-layer FFN with non-linear activation, and residual connections. By employing a vectorized analysis of the model's weights and a squared Frobenius norm loss function, they derived the gradients for all weight matrices (Query, Key, Value, FFN, and Unembedding).
The analysis demonstrates that with standard Gaussian initialization (e.g., He or Xavier), the training loss decreases at a linear rate. The convergence speed is mathematically tied to the singular values of the attention layer's output.
The Role of Residual Connections: Preventing Rank Collapse
A key insight of this work is the "rank collapse" problem inherent in softmax attention. The softmax operation tends to produce low-rank matrices, which can lead to an extremely flat optimization landscape where tokens become indistinguishable.
Residual connections act as mathematical stabilizers. By adding the original input $X$ back to the attention output, they ensure the resulting matrix $Z(X)$ maintains full rank. This "ameliorates the ill-conditioning" of the system, keeping the minimum and maximum singular values within a range that supports efficient gradient descent.
Empirical Validation
To verify the theoretical findings, experiments were conducted on both synthetic data and the Jena Climate Dataset—a real-world multivariate time series task.
As shown in the empirical results, models without residual connections fail to converge effectively, with the loss flatlining early in the training process. In contrast, models with residual connections exhibit rapid, linear convergence to a global minimum. This confirms that residuals are not just a heuristic but a mathematical necessity for stable Transformer training.
Conclusion
This paper provides a rigorous foundation for the success of residual connections in Transformers. While the current proof focuses on single-head, single-layer models without Layer Normalization, it offers a clear path forward for analyzing more complex architectures. Future work will aim to incorporate Layer Normalization and multi-head dynamics into this convergence framework.
Summary updated