In theory, deep Transformers should be victims of their own success. As a sequence of tokens passes through layer after layer of self-attention, the tokens exert an attractive force on one another. Mathematically, this looks like a system of particles gravitating toward a single point. If this "rank collapse" happened instantly, every token in a deep model would become identical, rendering the network's representations useless.
Yet, Transformers work. Recent research into the mean-field dynamics of these models explains this paradox through a phenomenon called dynamic metastability. Instead of collapsing immediately, tokens form stable, structured clusters that persist for an incredibly long time, providing the "breathing room" necessary for complex computation.
Tokens as Interacting Particles
To understand why Transformers don't immediately lose all their information, we have to look at them through the lens of physics. In A mathematical perspective on Transformers, Borjan Geshkovski, Yury Polyanskiy, Philippe Rigollet, and Cyril Letrouit propose treating tokens as a system of interacting particles moving on the surface of a high-dimensional sphere.
In this framework, the self-attention mechanism acts as a force that pulls particles together. Philippe Rigollet demonstrates in The Mean-Field Dynamics of Transformers that while the long-term destiny of these particles is "synchronization"—where all tokens merge into a single point—the path they take is far from a straight line.
The Metastable Buffer
The core discovery across this body of work is that tokens get "trapped" in intermediate states. In Dynamic metastability in the self-attention model, Geshkovski and colleagues prove that tokens form distinct clusters and remain in that configuration for an exponentially long duration.
This is the "metastability." In optimization, getting trapped in a local state is usually a failure. But for a Transformer, these long-lived intermediate states are a feature, not a bug. They allow the model to maintain diverse, semantically meaningful representations across many layers. Without this mechanism to slow down the transition toward global equilibrium, the model's representational capacity would vanish before it could finish a single forward pass.
Why Clusters Form
If tokens eventually want to be in one place, why do they stop at clusters first? The geometry of the attention mechanism provides the answer.
- Extreme Points: Albert Alcalde, Geshkovski, and Domènec Ruiz-Balet show in Attention's forward pass and Frank-Wolfe that the "hardmax" limit of attention is equivalent to a Frank-Wolfe optimization step. This causes token embeddings to gravitate toward the "extreme points" of their convex hull, effectively pushing them into corners rather than a single center.
- Geometric Packing: In Clustering in Causal Attention Masking, Nikita Karagodin, Polyanskiy, and Rigollet find that in causal models, early tokens in a sequence act as "nuclei." These early tokens serve as centers for cluster formation, a process they compare to the Rényi parking problem, where particles fill up space and prevent others from collapsing into them.
- Rotational Symmetries: Andrea Agazzi, Federico Pasqualotto, and Giuseppe Bruno argue in Emergence of meta-stable clustering in mean-field transformer models that these clusters are stabilized by emerging symmetries. These symmetries temporarily "lock" the tokens in place, preventing the final collapse into a single consensus point.
The Logarithmic Safety Net
How long does this stability last? The research suggests a direct link between the size of the model and the longevity of its representations. Agazzi and his team demonstrate that the duration of these metastable states is proportional to the logarithm of the number of tokens.
This provides a mathematical explanation for why models with larger context windows can be deeper: more tokens create a larger "logarithmic buffer," allowing the system to maintain complex, non-collapsed representations through a greater number of successive layers. By the time the particles would finally synchronize into a single, uninformative point—as explored by Polyanskiy and Rigollet in Synchronization of mean-field models on the circle—the forward pass is already complete.