Training massive neural networks is an expensive gamble. When scaling a model from millions to billions of parameters, practitioners traditionally have to re-tune critical hyperparameters like learning rates and initialization scales. If the learning rate is too high, the model diverges; if it is too low, training stagnates.
This tuning process is computationally prohibitive at scale. However, a growing body of research shows that we can bypass this entirely. By fundamentally altering how network weights and learning rates scale with model dimensions, we can achieve zero-shot hyperparameter transfer: finding the optimal configuration on a tiny, inexpensive proxy model and applying it directly to a massive production model.
The Scaling Bottleneck
To understand why hyperparameters normally break during scaling, consider a standard matrix multiplication layer, $h = Wx$, where $W$ is a weight matrix of size $d \times d$. In Standard Parameterization (SP), weights are typically initialized with a variance of $1/d$. This ensures that the total variance of the activations $h$ remains stable at $1$ at initialization, regardless of the width $d$.
The problem arises once training begins. When we compute a gradient update $\Delta W$, its magnitude depends on the width of the layer. Under standard gradient descent, a constant learning rate causes the subsequent change in activations, $\Delta h = \Delta W x$, to scale with the width $d$. As $d$ approaches infinity, these activation updates either explode or vanish.
As explored by Geiger et al. in Disentangling feature and lazy learning in deep neural networks: an empirical study, standard scaling choices inadvertently push wide networks into a "lazy training" regime where the model behaves like a fixed linear kernel, failing to learn rich representations. Conversely, Ba demonstrates in High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation that adjusting the learning rate relative to the network width allows the model to escape this lazy regime, actively adapting its internal features to the data distribution.
Stabilizing the Update Magnitude
The core insight behind solving this bottleneck is simple: the functional impact of a training update should be invariant to the size of the model. If a single gradient step modifies the network's internal representations by the same relative magnitude whether the width is 64 or 4096, then the optimal learning rate will remain completely stationary across scales.
This mechanism was formalized as Maximal Update Parameterization ($\mu$P) by Yang et al. in Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. Instead of treating all layers uniformly, $\mu$P scales both the weight initialization and the learning rate as explicit functions of each layer's input and output dimensions (its fan-in and fan-out).
For a hidden layer of width $d$:
- The weights are initialized such that feature variance is preserved.
- The learning rate is scaled inversely with the width (e.g., $\eta \propto 1/d$).
By enforcing these rules, the magnitude of the activation updates $\Delta h$ remains perfectly stable ($O(1)$) as width scales to infinity. The optimal learning rate landscape becomes invariant to width, allowing practitioners to run an exhaustive grid search on a cheap proxy model and transfer those exact parameters to a giant architecture.
Expanding Beyond Width
While $\mu$P originally targeted network width, recent work has expanded this scale-invariance to other geometric dimensions, such as depth and operator structure.
Depth Invariance
When networks grow deeper, the accumulation of residual connections can still cause signal explosion or vanishing features. To address this, Dey et al. introduced CompleteP in Don't be lazy: CompleteP enables compute-efficient deep transformers. By setting the residual scaling factor to unity and carefully balancing layerwise updates, the authors achieved stable hyperparameter transfer across both width and depth, allowing shallow models to serve as proxies for deep ones.
Taking a more geometric approach, Large et al. proposed Scalable Optimization in the Modular Norm. They introduced a recursively defined architectural norm that normalizes update vectors to a unit length within the network's structural geometry, rendering learning rates inherently invariant to both width and depth.
Operator and Spectral Scaling
Rather than relying strictly on coordinate-based dimension counting, alternative frameworks look at the mathematical operators themselves. Pethick proposes in Training Deep Learning Models with Norm-Constrained LMOs that utilizing operator norms—such as the spectral norm—within the update rules naturally preserves optimization stability. Because the spectral norm bounds the maximum amplification of a matrix regardless of its shape, bounding updates via this norm guarantees width-invariant hyperparameter scaling out of the box.
Practical Considerations: Numerical Health and Regularization
Implementing scale-invariant parameterization in real-world settings introduces low-level challenges, particularly regarding numerical precision and regularization.
Numerical Health
When scaling models to billions of parameters, scaling learning rates down by $1/d$ can cause weight updates to fall below the minimum representable value of lower-precision formats like FP16 or FP8, leading to underflow.
To reconcile hyperparameter stability with numerical safety, Blake et al. developed u-$\mu$P in u-$\mu$P: The Unit-Scaled Maximal Update Parametrization. This approach combines $\mu$P scaling rules with unit scaling, dynamically shifting model scales so that activations, weights, and gradients maintain a variance of approximately $1$ throughout training. This prevents numerical overflow and underflow while preserving zero-shot transferability.
Regularization Scaling
Hyperparameters do not exist in a vacuum; optimization is heavily influenced by regularization techniques like Sharpness-Aware Minimization (SAM), which perturbs weights to find flatter minima.
In $\boldsymbol{\mu}\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling, Haas et al. demonstrated that transferring an optimal learning rate is insufficient if the regularization parameters mismatch at scale. The authors introduced layer-specific exponents for the SAM perturbation radius, ensuring that both the weight updates and the regularization geometry remain stationary across scales.
Ubiquitous Transferability
The mathematical principles underlying scale-invariant parameterization are highly generalizable and extend far beyond standard multi-layer perceptrons or language models.
- Generative Models: Zheng et al. successfully extended $\mu$P to diffusion architectures in Scaling Diffusion Transformers Efficiently via $\mu$P, proving that cross-attention and adaptive layer normalization remain stable under these scaling laws.
- Scientific Computing: For complex physics modeling, Li et al. developed Maximal Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators, enabling zero-shot transfer across varying numbers of Fourier modes.
- Meta-Learning: Even learned optimizers—neural networks trained to optimize other neural networks—benefit from this framework. As shown by Thérien et al. in $\mu$LO: Compute-Efficient Meta-Generalization of Learned Optimizers, applying width-independent parameterization to the optimizer's own architecture prevents activation explosions, allowing it to generalize seamlessly to target models far larger than those it encountered during meta-training.
By ensuring that the internal dynamics of neural networks remain invariant to their physical size, scale-invariant parameterization transforms hyperparameter tuning from an expensive guessing game into a predictable, small-scale engineering task.