The Phased Training Dynamics of Transformer Attention

This direction is split into other directions: two-phase training dynamics. Its last version stays readable and citable here.

Version 1 of 139 papers55 conceptsfrozen 2026-10-04

While the training of deep neural networks is often visualized as a smooth descent down a loss landscape, recent theoretical and empirical evidence suggests that Transformers learn through a highly structured, multi-stage progression. Rather than optimizing all components simultaneously, these models exhibit a "staged convergence" where different functional units—such as feed-forward networks (FFNs), value matrices, and attention heads—activate and refine in a predictable sequence.

The Foundation Phase: Learning What Before Where

In the earliest stages of training, Transformers do not immediately begin "attending" to complex relationships. Instead, they focus on building a foundational representation of the data. Bhavya Kailkhura, Hongru Yang, and colleagues have shown that FFN layers typically align with target signals first to achieve basic classification while the attention mechanism remains nearly uniform.

This "feature-first" priority is a recurring theme. Zheng-An Chen and Tao Luo observe that under small initialization regimes, the key and query matrices remain dormant while the model optimizes value and feed-forward weights. Similarly, Yuchen Li and Yuanzhi Li found that value matrices initially capture broad semantic clusters (like word topics) before the attention mechanism evolves to refine the specific relationships between those clusters.

The intuition here is simple: a model cannot effectively learn where to look (attention) until it has a basic understanding of what it is looking at (features).

The Hidden Progress of Loss Plateaus

One of the most frustrating aspects of training Transformers is the "loss plateau"—long periods where the training error refuses to budge. However, researchers like Pulkit Gopalani and Wei Hu argue that these plateaus mask significant "hidden progress." During these phases, the model is often struggling with representation collapse or "repetition bias," but it is internally refining its attention maps.

Aditya Varre and Nicolas Flammarion demonstrate that simpler heuristics, like unigram or bigram patterns, act as "near-stationary points" that trap the gradient. The model remains in these plateaus until it suddenly "clicks," escaping the simpler heuristic to master more complex, multi-token dependencies. This transition is often described by Yedi Zhang and Andrew M Saxe as "saddle-to-saddle dynamics," where the model jumps from one performance plateau to the next as it discovers new dimensions of the task.

The Emergence Phase: Sharpening and Linking

Once the foundational features are established, the model enters a phase of rapid structural emergence. Heejune Sheen and Zhuoran Yang characterize this as a transition from a "latent warm-up" to a "rapid emergence" where task-specific weights lock in and cross-head interference vanishes.

In this stage, the attention mechanism transforms from a broad, exploratory tool into a precise filter. Ruiquan Huang and Yingbin Liang describe this as a shift from feature alignment to "margin maximization," where the model sharpens its focus to increase classification confidence. Hongru Yang and Zhangyang Wang show that this is when the model learns "correspondence structures"—linking disparate signals into a unified association.

This sharpening is not always monotonic. Yuandong Tian and colleagues observed a "sparse-then-dense" pattern: attention first becomes highly sparse to capture salient, high-frequency tokens, and later becomes denser to incorporate subtler, lower-frequency information.

Specialization and Reinforcement Loops

In the final stages of training, the model moves toward high-precision specialization. Umberto Nanni found that attention heads spontaneously organize into "wavelet-like" frameworks, specializing in different resolutions of positional and frequency data.

This specialization is often driven by a mutual reinforcement loop. As noted by Jiarui Jiang and Taiji Suzuki, as the attention mechanism sharpens its focus on "signal" tokens, the features of those tokens are learned more rapidly. This, in turn, makes the tokens even easier for the attention mechanism to identify, creating a virtuous cycle that drives the training loss toward zero.

By understanding these stages—from foundational feature learning to the sudden "click" of emergence and the final refinement of specialized heads—researchers are beginning to demystify why Transformers are so effective at navigating high-dimensional, noisy data.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_phased_training_dynamics_of_transfor_24dcb188,
  title = {The Phased Training Dynamics of Transformer Attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-phased-training-dynamics-of-transformer-attention/rdr_24dcb188a04d4689bc2f6c5c484c1dcb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_phased_training_dynamics_of_transfor_24dcb188_v1,
  title = {The Phased Training Dynamics of Transformer Attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-phased-training-dynamics-of-transformer-attention/rdr_24dcb188a04d4689bc2f6c5c484c1dcb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}