Abstract
Modern Transformers consistently learn to process language in structured, generalizable ways, a phenomenon often attributed to inductive bias. This paper identifies parameter norm growth as a critical, yet overlooked, inductive bias inherent in the training process. We demonstrate that the $\ell_2$ norm of Transformer parameters grows at a rate proportional to $\sqrt{t}$ during training. This growth drives the model toward a "saturated" state where continuous activations approximate discrete logic gates, facilitating the emergence of structured operations such as local attention and global averaging.
Introduction
Transformers are universal approximators with the capacity to represent a vast array of functions. However, during training via Gradient Descent, they tend to converge toward solutions that exhibit specific linguistic structures. This research investigates how the increasing magnitude of weights—rather than just their final values—shapes the model's internal logic. By analyzing the trajectory of parameter norms, we show that the training process naturally discretizes the model, aligning its computational properties with the requirements of formal grammars and syntax.
Methodology
The study employs a multi-faceted approach combining empirical tracking and theoretical analysis:
- Empirical Tracking: The authors monitored the $\ell_2$ norm of parameters across 104 checkpoints of the T5-base model and trained custom Transformers on Wikitext-2 and Penn Treebank (PTB) to observe growth patterns.
- Saturation Analysis: Internal representations were compared against a "saturated" limit where weights are scaled by a constant $c \to \infty$.
- Theoretical Framework: Using homogeneity theory, the authors proved that Transformers are "approximately 1-homogeneous," explaining the observed $\sqrt{t}$ growth rate. They further demonstrated that as norms diverge, the network converges to a "saturated network" where nonlinearities (Softmax, Sigmoid) function as discrete step functions.
Key Insights: The Saturation Bias
The primary finding is that norm growth is not a failure of convergence but a purposeful trajectory toward a stable directional solution. As weight magnitudes increase, the "sharpness" of Softmax and Sigmoid functions also increases. This effectively discretizes the model, reducing its theoretical power (it is no longer Turing-complete) but enhancing its ability to perform discrete logic.
This saturation leads to the emergence of two distinct types of attention heads:
- Argmax Heads: These focus on a single, most-relevant token, enabling local focus.
- Mean Heads: These average information across the entire sequence, allowing the model to "count" token occurrences—a fundamental requirement for recognizing formal languages.
Empirical Results
The researchers confirmed these theoretical predictions through several key observations regarding the training dynamics of T5 and other models.
As shown in Figure 2, parameters exhibit a consistent growth rate of $\sqrt{t}$. Simultaneously, the direction of the weight vector stabilizes rapidly, with cosine similarity between subsequent checkpoints approaching 1.0. Furthermore, pretrained models like BERT, RoBERTa, and T5 show high saturation levels (cosine similarity $>0.9$ with their discretized versions) in final layers, whereas randomly initialized models do not. In practice, attention heads in trained models tend to exhibit a bimodal distribution—either attending to a single token or distributing attention uniformly—confirming the emergence of "Argmax" and "Mean" behaviors.
Discussion and Future Work
While this study highlights the importance of norm growth, several questions remain. The interaction between aggressive weight decay and saturation needs further exploration, as standard regularization settings do not halt growth but might influence its rate. Additionally, while the paper focuses on the saturated limit ($c \to \infty$), real-world models operate in a "semi-saturated" regime. Future work is needed to characterize the specific computational properties of this intermediate state and the role of "intermediate" attention heads that do not strictly follow the Argmax or Mean modes.
Summary updated