Standard Transformers are notoriously difficult to analyze mathematically. The primary culprit is the softmax function in the self-attention layer:
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V$$
The softmax operator introduces an exponential non-linearity and a global normalization step that couples all tokens across a sequence. This couples the optimization dynamics, destroying the clean algebraic properties that mathematicians rely on to prove convergence, expressivity, or generalization behavior.
To make these networks mathematically tractable, a large body of theoretical literature relies on architectural simplifications—most notably, replacing softmax attention with linear attention. By stripping away softmax, multi-head structures, or feed-forward networks (MLPs), researchers can open the "black box" of mechanisms like In-Context Learning (ICL). However, this creates a profound tension: do the elegant proofs derived for these simplified models accurately explain the behavior of production-scale, non-linear Transformers?
Why Linear Attention Makes Theory Solvable
To understand why theorists rely on linear attention, consider what happens when you remove the softmax operator. The attention equation simplifies to a pure matrix product:
$$A = (QK^T)V$$
Because matrix multiplication is associative, we can rewrite this computation by shifting the parentheses:
$$A = Q(K^TV)$$
This simple algebraic shift changes the computational geometry entirely. In standard softmax attention, we must compute the $N \times N$ similarity matrix $QK^T$ first, which scales quadratically with the sequence length $N$. In linear attention, we can compute $K^TV$ first. If $K$ and $V$ have a hidden dimension $d$, then $K^TV$ is a static $d \times d$ matrix, regardless of how long the sequence grows.
This associativity allows researchers to treat the forward pass of a Transformer as an explicit statistical or optimization algorithm. For instance, Ahn et al. and Xiang Cheng demonstrated in Transformers learn to implement preconditioned gradient descent for in-context learning that when a linear Transformer is trained on linear regression tasks, its weights converge to a state that executes iterations of preconditioned gradient descent during its forward pass.
Similarly, Zhang et al. proved in Trained Transformers Learn Linear Models In-Context that a single-layer linear self-attention model naturally converges via gradient flow to implement Ordinary Least Squares (OLS). Without the softmax non-linearity, the prediction function becomes a clean quadratic form of the model parameters, allowing the direct application of matrix factorization theory to study how attention layers learn statistical tasks (Zheng et al.). It allows researchers to show how linear attention can even emulate second-order optimization methods like Newton's Method entirely within its forward pass (Giannou et al.).
The Non-Linear Approximation Gap
While linear simplifications yield beautiful proofs, they introduce a significant "approximation gap." Softmax is not just a mathematical nuisance; it fundamentally alters how a model processes information.
The core difference lies in selection versus averaging. Linear attention computes a weighted average across all tokens based on unnormalized inner products. Softmax attention, due to its exponential nature, acts as a competitive, "winner-take-all" mechanism. It allows a model to perfectly isolate a single, highly relevant token out of thousands—a property crucial for tasks like retrieval or tracking variable bindings. As noted in The Transformer Cookbook, this discrepancy between idealized token selection and practical weighted averaging means that theoretical computational bounds may not strictly hold in real-world systems.
Furthermore, Yedi Zhang et al. point out in Training Dynamics of In-Context Learning in Linear Attention that omitting the softmax operation fails to capture how non-linear normalization affects the convergence and stability of in-context learning during training. When softmax is present, the optimization landscape changes drastically. Elvis Dohmatob, in Understanding Softmax Attention Layers:\ Exact Mean-Field Analysis on a Toy Problem, highlights that prior theoretical analyses relying on pointwise simplifications completely miss the global normalization properties of softmax, which are essential for finding "needle in a haystack" tokens during optimization.
Middle Grounds and Taylor Expansions
To bridge this gap, some researchers explore intermediate mathematical representations that retain some non-linearity while preserving tractability. One common approach is a linearized softmax approximation using a first-order Taylor expansion (i.e., approximating $e^x \approx 1 + x$).
For example, Demir et al. utilize this technique in Optimal Attention Temperature Enhances In-Context Learning under Distribution Shift to derive closed-form solutions for generalization errors under distribution shifts. However, the authors note a critical limitation: these linearized approximations break down in regimes characterized by extremely sharp attention distributions, where the weights concentrate heavily on a tiny subset of tokens.
On the empirical side, architectures like MB-TaylorFormer, proposed by Kaihao Zhang in MB-TaylorFormer: Multi-branch Efficient Transformer Expanded by Taylor Formula for Image Dehazing, use Taylor expansions to achieve linear computational complexity ($O(n)$) while maintaining a global receptive field. Yet, as a general rule, these approximations risk introducing errors in global context modeling compared to full softmax.
Missing Components Beyond Softmax
The simplification gap extends beyond the attention operation itself. To make proofs mathematically tractable, theoretical frameworks frequently strip away other standard components of the Transformer architecture:
- Feed-Forward Networks (MLPs): Many mechanistic and circuit-based theories assume attention-only models (Song et al.). Yet, MLPs introduce vital non-linear representational shifts that allow models to perform non-linear in-context tasks (Sun et al.) and manage out-of-context reasoning (Huang et al.).
- Layer Normalization and Residuals: Standard proofs often omit LayerNorm or assume specific initializations to remain tractable (Wen et al.). As Musat et al. argue, omitting these features means theories cannot account for the messier, more complex dynamics that govern how features like induction heads emerge in production-scale models.
- Multi-Head and Untied Weights: Theorists often assume a single attention head or tied query-key weights ($W_Q = W_K$) to simplify the optimization landscape (Boncoraglio et al.). This overlooks the complex interference patterns and multi-dimensional routing that occur when multiple heads interact.
Ultimately, the study of linear attention simplifications reveals a fundamental trade-off in deep learning theory. Linearization provides the clean, associative mathematics necessary to prove why a Transformer can act as an implicit optimizer or data-driven computer. But understanding the full power of modern large language models requires acknowledging the non-linear gap—the competitive selection, representational shifts, and structural interactions that only emerge when all the pieces of the architecture work together.