Zero Shot Hyperparameter Transfer Across Model Scales

263 concepts132 papers1 proposal

Training massive neural networks is an expensive gamble. When scaling a model from millions to billions of parameters, practitioners traditionally have to re-tune critical hyperparameters like learning rates and initialization scales. If the learning rate is too high, the model diverges; if it is too low, training stagnates.

This tuning process is computationally prohibitive at scale. However, a growing body of research shows that we can bypass this entirely. By fundamentally altering how network weights and learning rates scale with model dimensions, we can achieve zero-shot hyperparameter transfer: finding the optimal configuration on a tiny, inexpensive proxy model and applying it directly to a massive production model.

The Scaling Bottleneck

To understand why hyperparameters normally break during scaling, consider a standard matrix multiplication layer, $h = Wx$, where $W$ is a weight matrix of size $d \times d$. In Standard Parameterization (SP), weights are typically initialized with a variance of $1/d$. This ensures that the total variance of the activations $h$ remains stable at $1$ at initialization, regardless of the width $d$.

The problem arises once training begins. When we compute a gradient update $\Delta W$, its magnitude depends on the width of the layer. Under standard gradient descent, a constant learning rate causes the subsequent change in activations, $\Delta h = \Delta W x$, to scale with the width $d$. As $d$ approaches infinity, these activation updates either explode or vanish.

As explored by Geiger et al. in Disentangling feature and lazy learning in deep neural networks: an empirical study, standard scaling choices inadvertently push wide networks into a "lazy training" regime where the model behaves like a fixed linear kernel, failing to learn rich representations. Conversely, Ba demonstrates in High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation that adjusting the learning rate relative to the network width allows the model to escape this lazy regime, actively adapting its internal features to the data distribution.

Stabilizing the Update Magnitude

The core insight behind solving this bottleneck is simple: the functional impact of a training update should be invariant to the size of the model. If a single gradient step modifies the network's internal representations by the same relative magnitude whether the width is 64 or 4096, then the optimal learning rate will remain completely stationary across scales.

This mechanism was formalized as Maximal Update Parameterization ($\mu$P) by Yang et al. in Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. Instead of treating all layers uniformly, $\mu$P scales both the weight initialization and the learning rate as explicit functions of each layer's input and output dimensions (its fan-in and fan-out).

For a hidden layer of width $d$:

  • The weights are initialized such that feature variance is preserved.
  • The learning rate is scaled inversely with the width (e.g., $\eta \propto 1/d$).

By enforcing these rules, the magnitude of the activation updates $\Delta h$ remains perfectly stable ($O(1)$) as width scales to infinity. The optimal learning rate landscape becomes invariant to width, allowing practitioners to run an exhaustive grid search on a cheap proxy model and transfer those exact parameters to a giant architecture.

Expanding Beyond Width

While $\mu$P originally targeted network width, recent work has expanded this scale-invariance to other geometric dimensions, such as depth and operator structure.

Depth Invariance

When networks grow deeper, the accumulation of residual connections can still cause signal explosion or vanishing features. To address this, Dey et al. introduced CompleteP in Don't be lazy: CompleteP enables compute-efficient deep transformers. By setting the residual scaling factor to unity and carefully balancing layerwise updates, the authors achieved stable hyperparameter transfer across both width and depth, allowing shallow models to serve as proxies for deep ones.

Taking a more geometric approach, Large et al. proposed Scalable Optimization in the Modular Norm. They introduced a recursively defined architectural norm that normalizes update vectors to a unit length within the network's structural geometry, rendering learning rates inherently invariant to both width and depth.

Operator and Spectral Scaling

Rather than relying strictly on coordinate-based dimension counting, alternative frameworks look at the mathematical operators themselves. Pethick proposes in Training Deep Learning Models with Norm-Constrained LMOs that utilizing operator norms—such as the spectral norm—within the update rules naturally preserves optimization stability. Because the spectral norm bounds the maximum amplification of a matrix regardless of its shape, bounding updates via this norm guarantees width-invariant hyperparameter scaling out of the box.

Practical Considerations: Numerical Health and Regularization

Implementing scale-invariant parameterization in real-world settings introduces low-level challenges, particularly regarding numerical precision and regularization.

Numerical Health

When scaling models to billions of parameters, scaling learning rates down by $1/d$ can cause weight updates to fall below the minimum representable value of lower-precision formats like FP16 or FP8, leading to underflow.

To reconcile hyperparameter stability with numerical safety, Blake et al. developed u-$\mu$P in u-$\mu$P: The Unit-Scaled Maximal Update Parametrization. This approach combines $\mu$P scaling rules with unit scaling, dynamically shifting model scales so that activations, weights, and gradients maintain a variance of approximately $1$ throughout training. This prevents numerical overflow and underflow while preserving zero-shot transferability.

Regularization Scaling

Hyperparameters do not exist in a vacuum; optimization is heavily influenced by regularization techniques like Sharpness-Aware Minimization (SAM), which perturbs weights to find flatter minima.

In $\boldsymbol{\mu}\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling, Haas et al. demonstrated that transferring an optimal learning rate is insufficient if the regularization parameters mismatch at scale. The authors introduced layer-specific exponents for the SAM perturbation radius, ensuring that both the weight updates and the regularization geometry remain stationary across scales.

Ubiquitous Transferability

The mathematical principles underlying scale-invariant parameterization are highly generalizable and extend far beyond standard multi-layer perceptrons or language models.

By ensuring that the internal dynamics of neural networks remain invariant to their physical size, scale-invariant parameterization transforms hyperparameter tuning from an expensive guessing game into a predictable, small-scale engineering task.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 128 papers
  2. Version 1 · 2026-10-04 · 273 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_maximal_update_parameterization_99534441,
  title = {maximal update parameterization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/maximal-update-parameterization/rdr_99534441e3874228b0c28ab4dcafda20},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 2, frozen text):

@misc{lacuna_maximal_update_parameterization_99534441_v2,
  title = {maximal update parameterization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/maximal-update-parameterization/rdr_99534441e3874228b0c28ab4dcafda20/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake