Superiority of Multi-Head Attention: A Theoretical Study in Shallow Transformers in In-Context Linear Regression

Yingqian Cui, Jie Ren, Pengfei He, Hui Liu, Jiliang Tang, Yue XingAISTATS 2025 · OpenReview

Abstract

We present a theoretical analysis of the performance of transformer with softmax attention in in-context learning with linear regression tasks. While the existing theoretical literature predominantly focuses on providing convergence upper bounds to show that trained transformers with single-/multi-head attention can obtain a good in-context learning performance, our research centers on comparing the exact convergence of single- and multi-head attention more rigorously. We conduct an exact theoretical analysis to demonstrate that multi-head attention with a substantial embedding dimension performs better than single-head attention. When the number of in-context examples $D$ increases, the prediction loss using single-/multi-head attention is in $O(1/D)$, and the one for multi-head attention has a smaller multiplicative constant. In addition to the simplest data distribution setting, our technical framework in calculating the exact convergence further facilitates studying more scenarios, e.g., noisy labels, local examples, correlated features, and prior knowledge. We observe that, in general, multi-head attention is preferred over single-head attention. Our results verify the effectiveness of the design of multi-head attention in the transformer architecture.

Introduction

In-Context Learning (ICL)—the ability of a model to learn a new task from just a few examples in a prompt without any parameter updates—is a hallmark of modern Transformers. While empirical evidence has long suggested that multi-head attention (MHA) is superior to single-head attention (SHA) for these tasks, the theoretical justification has remained elusive. Previous research mostly focused on providing "upper bounds," which show that a model can perform well but fail to explain the specific advantage of multiple heads. This study addresses this gap by deriving exact convergence rates for both SHA and MHA in a shallow Transformer setting, specifically for in-context linear regression.

Methods and Theoretical Framework

The study analyzes a one-layer Transformer with softmax attention. The ICL process is modeled as a linear regression task where the model receives a prompt containing $D$ pairs of $(x_i, y_i)$ and must predict $y_q$ for a new query $x_q$.

To move beyond loose upper bounds, the authors utilize Taylor expansions on attention scores to calculate the exact prediction risk (Mean Squared Error). They identify optimal configurations for the weight matrices ($W_K, W_Q, W_V, W_{out}$) that minimize prediction loss. The framework is tested against synthetic data generation models, including standard Gaussian distributions, Student’s t-distributions, and Gaussian Mixture Models (GMM), to ensure robustness. Additionally, the authors used LLaMA model token embeddings to estimate "intrinsic dimension" via Local Principal Components Analysis (LPCA), justifying their theoretical focus on dimensionality.

Key Insights: Why More Heads are Better

The "Negative Weight" Advantage

In single-head attention, all attention scores are non-negative. MHA, however, can combine scores from different heads. By subtracting one head's output from another, the Transformer can effectively assign "negative" weights to certain examples. This added flexibility allows for a much more precise fit of the underlying linear function.

The $O(1/D)$ Convergence Rule

Both SHA and MHA see their prediction error vanish at a rate of $1/D$, where $D$ is the number of examples in the prompt. However, MHA possesses a smaller multiplicative constant, meaning it learns more efficiently from the same number of examples.

Squared error convergence as a function of in-context examples for different numbers of attention heads.

As shown in the figure above, increasing the number of heads significantly accelerates the reduction of squared error as the number of in-context examples grows, particularly in the early stages of learning.

Intrinsic vs. Ambient Dimension

A common belief is that more heads are always better if the embedding dimension is large. This research argues that the optimal number of heads is actually constrained by the intrinsic dimension ($d$) of the data. If the data is low-dimensional, adding heads beyond a certain point (where the dimension per head $p/h < d$) can actually degrade performance.

Key Results

Theoretical formulas and simulations consistently show MHA achieving lower Mean Squared Error (MSE) than SHA across various dimensions. The superiority of MHA is particularly evident when comparing 1-head and 2-head configurations across different prompt lengths and data dimensions.

Comparison of MSE for 1-head and 2-head attention across varying intrinsic dimensions $d$ and prompt lengths $D$.

In the figure above, the 2-head configuration (solid lines) consistently maintains a lower MSE than the 1-head configuration (dashed lines) across all tested intrinsic dimensions ($d$). Furthermore, the gap between SHA and MHA remains significant even as the number of in-context examples ($D$) increases.

Conclusion and Future Work

This study provides a rigorous theoretical foundation for the design of multi-head attention, demonstrating its efficiency in learning from in-context examples. While the research focuses on a single-layer Transformer and linear regression, it sets the stage for future analysis of deeper architectures and non-linear tasks. The findings suggest that the effectiveness of MHA is not merely a result of increased parameter count, but a fundamental architectural advantage in processing and weighing contextual information.

Summary updated

Abstract

We present a theoretical analysis of the performance of transformer with softmax attention in in-context learning with linear regression tasks. While the existing theoretical literature predominantly focuses on providing convergence upper bounds to show that trained transformers with single-/multi-head attention can obtain a good in-context learning performance, our research centers on comparing the exact convergence of single- and multi-head attention more rigorously. We conduct an exact theoretical analysis to demonstrate that multi-head attention with a substantial embedding dimension performs better than single-head attention. When the number of in-context examples $D$ increases, the prediction loss using single-/multi-head attention is in $O(1/D)$, and the one for multi-head attention has a smaller multiplicative constant. In addition to the simplest data distribution setting, our technical framework in calculating the exact convergence further facilitates studying more scenarios, e.g., noisy labels, local examples, correlated features, and prior knowledge. We observe that, in general, multi-head attention is preferred over single-head attention. Our results verify the effectiveness of the design of multi-head attention in the transformer architecture.
BibTeX
@inproceedings{cui2025superiority,
  author = {Yingqian Cui and Jie Ren and Pengfei He and Hui Liu and Jiliang Tang and Yue Xing},
  title = {{Superiority of Multi-Head Attention: A Theoretical Study in Shallow Transformers in In-Context Linear Regression}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
  url = {https://openreview.net/forum?id=lF0MrIGQlo},
}
Figure