On the Convergence of Gradient Descent on Learning Transformers with Residual Connections

Zhen Qin, Jinxin Zhou, Zhihui Zhu

Abstract

Transformer models have become foundational in AI, yet the theoretical understanding of their training dynamics remains incomplete. This paper investigates the convergence of gradient descent on a structurally complete single-layer Transformer. We prove that under appropriate initialization, gradient descent achieves a linear convergence rate. Crucially, we identify that residual connections are vital for maintaining the rank of the attention output, thereby preventing ill-conditioning and ensuring optimization stability.

Introduction

Despite the empirical dominance of Transformers, mathematical analysis often simplifies the architecture by isolating components like self-attention or feedforward networks (FFNs). This study bridges that gap by analyzing the interplay between self-attention, FFNs, and residual connections. The central question is how these components collectively influence the optimization landscape to reach a global minimum.

Theoretical Framework

The researchers analyzed a complete Transformer block consisting of softmax self-attention, a two-layer FFN with non-linear activation, and residual connections. By employing a vectorized analysis of the model's weights and a squared Frobenius norm loss function, they derived the gradients for all weight matrices (Query, Key, Value, FFN, and Unembedding).

Architecture of the single-layer Transformer block analyzed, including self-attention, feedforward network, and residual connections.

The analysis demonstrates that with standard Gaussian initialization (e.g., He or Xavier), the training loss decreases at a linear rate. The convergence speed is mathematically tied to the singular values of the attention layer's output.

The Role of Residual Connections: Preventing Rank Collapse

A key insight of this work is the "rank collapse" problem inherent in softmax attention. The softmax operation tends to produce low-rank matrices, which can lead to an extremely flat optimization landscape where tokens become indistinguishable.

Residual connections act as mathematical stabilizers. By adding the original input $X$ back to the attention output, they ensure the resulting matrix $Z(X)$ maintains full rank. This "ameliorates the ill-conditioning" of the system, keeping the minimum and maximum singular values within a range that supports efficient gradient descent.

Empirical Validation

To verify the theoretical findings, experiments were conducted on both synthetic data and the Jena Climate Dataset—a real-world multivariate time series task.

Comparison of training loss convergence over iterations for Transformers with and without residual connections.

As shown in the empirical results, models without residual connections fail to converge effectively, with the loss flatlining early in the training process. In contrast, models with residual connections exhibit rapid, linear convergence to a global minimum. This confirms that residuals are not just a heuristic but a mathematical necessity for stable Transformer training.

Conclusion

This paper provides a rigorous foundation for the success of residual connections in Transformers. While the current proof focuses on single-head, single-layer models without Layer Normalization, it offers a clear path forward for analyzing more complex architectures. Future work will aim to incorporate Layer Normalization and multi-head dynamics into this convergence framework.

Summary updated

Abstract

Transformer models have emerged as fundamental tools across various scientific and engineering disciplines, owing to their outstanding performance in diverse applications. Despite this empirical success, the theoretical foundations of Transformers remain relatively underdeveloped, particularly in understanding their training dynamics. Existing research predominantly examines isolated components--such as self-attention mechanisms and feedforward networks--without thoroughly investigating the interdependencies between these components, especially when residual connections are present. In this paper, we aim to bridge this gap by analyzing the convergence behavior of a structurally complete yet single-layer Transformer, comprising self-attention, a feedforward network, and residual connections. We demonstrate that, under appropriate initialization, gradient descent exhibits a linear convergence rate, where the convergence speed is determined by the minimum and maximum singular values of the output matrix from the attention layer. Moreover, our analysis reveals that residual connections serve to ameliorate the ill-conditioning of this output matrix, an issue stemming from the low-rank structure imposed by the softmax operation, thereby promoting enhanced optimization stability. We also extend our theoretical findings to a multi-layer Transformer architecture, confirming the linear convergence rate of gradient descent under suitable initialization. Empirical results corroborate our theoretical insights, illustrating the beneficial role of residual connections in promoting convergence stability.
BibTeX
@misc{qin2025convergence,
  author = {Zhen Qin and Jinxin Zhou and Zhihui Zhu},
  title = {{On the Convergence of Gradient Descent on Learning Transformers with Residual Connections}},
  year = {2025},
  eprint = {2506.05249},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.05249},
}
Figure