When we train a Transformer, it can feel like a black box optimizing millions of weights simultaneously. However, recent mathematical analyses of gradient flow reveal a highly structured, step-by-step choreography. Instead of learning everything at once, attention-based models undergo a distinct multi-phase trajectory: they begin by building a coarse map of the data while keeping attention uniform, and then ignite a powerful, self-reinforcing feedback loop between attention weights and feature representations.
Understanding this phased progression provides a mechanistic explanation for how Transformers select sparse tokens, why their capabilities often emerge suddenly, and how they avoid shortcut learning.
Phase One: Representation Alignment with Dormant Attention
Why doesn't a Transformer immediately start picking out specific tokens at the beginning of training? The answer lies in the initialization and the early gradients. Under standard regimes—especially small initialization—the key and query matrices remain nearly static at first. As Zheng-An Chen et al. demonstrate, this effectively keeps the attention mechanism dormant during the initial training phase, forcing the model to focus its optimization efforts on the outer layers, value matrices, and feed-forward networks (FFNs).
During this first stage, the model learns what the features are before figuring out where they are or how they relate to one another. As Bhavya Kailkhura et al. show, the feed-forward layers initially align with target signals to achieve basic classification while the attention distribution remains uniform.
Initial Phase:
[Input Tokens] ---> [ Uniform Attention ] ---> [ FFN / Value Alignment ] ---> Coarse Clusters
This behavior is mirrored across different components of the architecture:
- Value Matrices: Yuchen Li et al. prove that value matrices prioritize global context aggregation early on, capturing broad semantic or topic clusters before any specialized token-to-token focus develops.
- Neuron Alignment: Hongru Yang et al. show that individual neurons first align to raw input signals, transforming initially non-separable data into a well-separated representation before the attention mechanism begins linking those signals together.
Phase Two: The Attention-Feature Feedback Loop
Once the foundational features are coarsely aligned, the attention mechanism awakens. This transition marks the beginning of a highly dynamic second phase driven by a "mutual promotion" synergy between routing (attention) and representation (embeddings).
As mathematically proven by Haoye Lu et al., a positive feedback loop exists between a token's attention weight and its internal embedding norm. The mechanism operates as a cycle:
$$\text{Attention Score Increases} \implies \text{Embedding Norm Grows} \implies \text{Attention Accelerates Further}$$
Jiarui Jiang et al. describe this as a mutual reinforcement loop between the attention mechanism and feature learning components. As the attention weights focus more on signal-bearing tokens, the feature weights receive cleaner gradients and learn the signal more effectively. This refined feature representation, in turn, provides stronger feedback to further sharpen the attention focus.
┌────────────────────────────────────────┐
▼ │
[Sharper Attention Focus] ──► [Cleaner Signal to FFN] ──► [Stronger Feature Weights]
This feedback loop explains why Transformers are remarkably efficient at sparse token selection. Zixuan Wang et al. show that this trajectory allows the attention mechanism to rapidly isolate a sparse subset of relevant information from high-dimensional background noise—a capability that fully-connected networks fundamentally lack without an exponential increase in parameters.
Phase Three: Margin Maximization and Emergence
After the feedback loop isolates the relevant tokens, the training dynamics shift into a final stabilization phase. Ruiquan Huang et al. demonstrate that feature alignment and confidence maximization are decoupled temporal processes. Once the attention mechanism successfully maps input sequences into a linearly separable space, the model transitions to a phase where the linear layers optimize and maximize the classification margin, moving the model from a merely feasible solution to a robust, confident one.
This sequential convergence—moving from feature alignment to attention concentration, and finally to joint refinement—explains how the model progressively locks in both the features and the structural relationships of a task (Chenyang Zhang et al.).
Why Capabilities "Emerge" Suddenly
This phased, non-linear progression provides a elegant explanation for the phenomenon of "emergent capabilities," where a model's performance suddenly jumps after a long period of stagnation.
Because the mutual reinforcement loop between attention and feed-forward alignment is highly non-linear, the model can appear to be making zero progress during Phase One while it quietly builds its foundational feature clusters. As Nicolas Zucchet et al. demonstrate, once the attention weights begin to concentrate on key tokens, the feedback loop ignites, causing a rapid, exponential improvement in performance that looks like a sudden breakthrough.
Designing Around the Dynamics
Recognizing that attention models naturally learn in stages has allowed researchers to design better training interventions and interpretability tools.
Overcoming Shortcuts
Standard joint training can sometimes cause models to latch onto superficial shortcuts (like background colors in images) before the attention mechanism can isolate the true object. To counteract this, Dipkamal Bhusal et al. introduced a sequential training procedure that enforces an initial phase dedicated exclusively to aligning a model's internal attention with a semantic mask before introducing classification objectives. This ensures feature selection is grounded in relevance before mapping to labels.
In vision tasks, this attention-first grounding often happens implicitly. Samy Jelassi et al. proved that Vision Transformers naturally develop spatial attention masks to filter out noise before refining feature representations for high accuracy. This mirrors clinical workflows, such as scanning a medical image globally before zooming in on specific lesions for joint refinement (Yanxin Yin et al.).
A Footprint in the Weights
Because features align progressively during training—with the most significant features stabilizing first—this dynamic leaves a permanent structural footprint. Gabriel Franco et al. discovered that the semantic features a model manipulates directly align with the top singular vectors of the attention head's weight matrices. This alignment is a mathematical consequence of minimizing attention loss while orthogonalizing features, providing a concrete, weight-based lens for mechanistic interpretability.
Key Foundational Work
On the Dynamics of Training Attention Models (2021) — contributed 2 concept elements to this direction.
Zoom-in-Net: Deep Mining Lesions for Diabetic Retinopathy Detection (2017) — contributed 1 concept elements to this direction.