Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent

William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, Noah Smith

TL;DR

Training Transformers with gradient descent causes their parameter norms to grow steadily over time, pushing the models toward a "saturated" state where activations behave like discrete step functions. This saturation acts as a crucial inductive bias, restricting the model's vast theoretical capacity to a set of simpler, linguistically relevant operations like counting and local selection.

What Problem Does This Paper Address?

Modern neural networks, especially Transformers, are massive overparameterized models. Theoretically, they are "universal approximators," meaning they have the capacity to learn almost any function. Yet, in practice, they generalize remarkably well to the specific, structured patterns of human language rather than just memorizing training data.

This suggests that the training process itself—typically a variant of Gradient Descent (GD)—imposes a hidden "inductive bias" that constrains what the model learns. The authors set out to characterize this bias by investigating norm growth: the empirical tendency for weights to increase in magnitude during training. They want to understand why this happens and how it fundamentally changes the model's internal representations.

Methods

The researchers combined large-scale empirical analysis with formal mathematical proofs:

  • Empirical Tracking: They analyzed 104 historical checkpoints from the pretraining of T5-base to document how parameter norms evolve.
  • Saturation Analysis: They used the concept of "saturation"—a state where a network's nonlinearities (like softmax or tanh) are pushed to their limits, effectively becoming discrete step functions. They proved that if parameter norms diverge at a uniform rate, the network mathematically converges to a "saturated" version of itself.
  • Homogeneity Theory: To explain the cause of norm growth, they analyzed Transformers through the lens of "homogeneous functions" (functions where scaling the input scales the output by a fixed power). They proved that Transformer encoders are approximately 1-homogeneous, which allows them to apply deep learning theory to explain training dynamics.
  • Attention Head Probing: They analyzed the attention patterns of saturated models to see what kind of discrete computations emerge.

Datasets

  • T5-base checkpoints: Used to observe real-world training dynamics.
  • Wikitext-2 & Penn Treebank (PTB): Used to train smaller Transformer language models from scratch for controlled experiments.
  • Brown Corpus: Used as a standard text source to evaluate the "saturation level" of pretrained models like BERT, RoBERTa, and XLNet.

Key Insights and Ideas

  • Saturation as a Feature, Not a Bug: Norm growth isn't just a side effect of training; it's the mechanism that creates inductive bias. By pushing the model toward saturation, GD simplifies the model's computational "language," making it more like a discrete automaton (like a counter machine) and less like a continuous, unconstrained computer.
  • Pretraining is a "Saturator": Pretrained models are significantly more saturated than their randomly initialized counterparts. This suggests that the "magic" of pretraining is partly about finding a direction in weight space that allows the model to become discrete and structured.
  • The $\sqrt{t}$ Growth Rate: The authors found that T5's parameter norm grows proportional to the square root of the training time ($\sqrt{t}$). They show this specific rate is a hallmark of "misaligned dynamics," where gradients aren't perfectly aligned with the weight direction—a common scenario in complex, scale-invariant architectures.
Figure 1: Top: Norm growth during T5 pretraining, with a coefficient r2 = 1.00. The right is broken down by layer. Bottom: cosine similarity between subsequent parameter checkpoints.

Key Results

  • High Saturation in Large Models: Pretrained models like T5 and XLNet show extremely high similarity ($\geq 0.9$ cosine similarity) to their saturated counterparts in later layers. Randomly initialized models, by contrast, show near-zero similarity.
  • Bimodal Attention: In saturated models, attention heads naturally split into two distinct functional types:
  • Argmax heads: These focus locally on a single position (e.g., the next word or a specific syntactic neighbor).
  • Mean heads: These average uniformly over the sequence, enabling the model to "count" occurrences of tokens.
  • Homogeneity Proofs: The paper provides formal proofs that Transformer components (LayerNorm, ReLU, Self-Attention) preserve "approximate homogeneity," providing a theoretical foundation for why these models exhibit consistent norm growth.
Figure 3: The distribution of attention heads in trained models is bimodal, showing a clear preference for either "argmax-like" (1 position) or "mean-like" (global) attention.

Limitations and Future Work

  • The "Semi-Saturated" Gap: While the paper characterizes fully saturated models, real-world Transformers are "semi-saturated." A precise formal characterization of this middle ground is still an open challenge.
  • Mysterious "Middle" Heads: In some models (like pre-norm Transformers), a small number of attention heads fall between the argmax and mean modes. Their specific function remains unknown.
  • Optimizer Specifics: The study focuses on T5 (AdaFactor) and AdamW. While they found that standard weight decay doesn't stop norm growth, different optimizers or extreme hyperparameter settings might alter these dynamics.
  • Simplifying Assumptions: The theoretical proofs for homogeneity often assume the absence of bias terms, though empirical results suggest the conclusions hold even when biases are present.

Summary updated

Abstract

The capacity of neural networks like the widely adopted transformer is known to be very high. Evidence is emerging that they learn successfully due to inductive bias in the training routine, typically a variant of gradient descent (GD). To better understand this bias, we study the tendency for transformer parameters to grow in magnitude ($\ell_2$ norm) during training, and its implications for the emergent representations within self attention layers. Empirically, we document norm growth in the training of transformer language models, including T5 during its pretraining. As the parameters grow in magnitude, we prove that the network approximates a discretized network with saturated activation functions. Such "saturated" networks are known to have a reduced capacity compared to the full network family that can be described in terms of formal languages and automata. Our results suggest saturation is a new characterization of an inductive bias implicit in GD of particular interest for NLP. We leverage the emergent discrete structure in a saturated transformer to analyze the role of different attention heads, finding that some focus locally on a small number of positions, while other heads compute global averages, allowing counting. We believe understanding the interplay between these two capabilities may shed further light on the structure of computation within large transformers.
BibTeX
@inproceedings{merrill2021effects,
  author = {William Merrill and Vivek Ramanujan and Yoav Goldberg and Roy Schwartz and Noah Smith},
  title = {{Effects of Parameter Norm Growth During Transformer Training: Inductive Bias from Gradient Descent}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.133},
}
Figure