Training modern deep learning models with billions of parameters is an incredibly expensive endeavor. When a single training run can cost millions of dollars, practitioners cannot afford to run traditional hyperparameter optimization loops—such as grid search or random sweeps—at full scale.
Ideally, developers would optimize hyperparameters like learning rate, batch size, and weight decay on a small, inexpensive "proxy model" and directly apply those settings to the final production architecture. However, under standard deep learning frameworks, optimal hyperparameters are tightly coupled to a model's specific dimensions. As Lorenzo Noci et al. demonstrate in Super Consistency of Neural Network Landscapes and Learning Rate Transfer, a learning rate that yields perfect convergence on a small model will often cause a larger version of the exact same architecture to diverge or stall.
The goal of cross-scale hyperparameter transfer is to break this coupling. By redesigning how networks are parameterized, how optimizers calculate updates, or how training schedules are structured, researchers have uncovered ways to make optimal hyperparameters invariant across scales.
Why Hyperparameters Shift with Scale
To understand why hyperparameters change as a model grows, consider what happens inside a standard neural network layer during a forward and backward pass.
When we scale a network's width (the number of hidden units $d$), we typically adjust the initialization variance of the weights by a factor of $1/\sqrt{d}$ (such as in Xavier or He initialization) to keep the variance of the activations stable during the forward pass. However, standard backpropagation does not treat the backward pass symmetrically. As the width increases, the collective matrix updates during gradient descent can cause the representation change in deeper layers to grow or shrink at rates proportional to the width.
If you scale up a model's width while keeping the learning rate constant, the magnitude of the weight updates can cause the network's activations to explode. Conversely, if you scale down the learning rate too aggressively to maintain stability, the network can fall into an under-parameterized regime where it fails to learn rich features. This scale sensitivity forces a costly trial-and-error process for every increase in model size.
Stabilizing the Infinite-Width Limit: $\mu$P
The primary breakthrough in resolving this width-scaling dilemma is Maximal Update Parameterization ($\mu$P), introduced by Ge Yang et al. in Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer.
The core intuition of $\mu$P is to scale both the initialization and the learning rates of individual layers as a function of their width. Instead of applying a uniform learning rate across the entire network, $\mu$P adjusts the learning rate of each layer dynamically (for example, scaling the learning rate of feature-learning layers by $1/d$). This precise calibration ensures that as the width approaches infinity, the feature representations and the magnitude of their updates remain stable ($O(1)$).
Standard Parameterization:
Small Model LR ───► Works well
Large Model LR ───► Diverges (Updates scale with width)
Maximal Update Parameterization (µP):
Small Proxy LR ───► [ µP Scaling Rules ] ───► Large Model LR (Zero-Shot Transfer)
By adopting $\mu$P, the loss landscape of a tiny proxy model becomes a reliable predictor of a massive model's behavior. This framework has proven robust across diverse settings. For instance, Saaketh Narayan et al. developed $\mu$nit Scaling in µnit Scaling: Simple and Scalable FP8 LLM Training, combining unit-variance principles with $\mu$P to enable seamless hyperparameter transfer in low-precision FP8 training without the need for complex dynamic scaling.
Scale-Invariant Optimizers and Geometry
While $\mu$P modifies the network's underlying parameterization, an alternative approach achieves cross-scale transfer by altering the optimizer or the geometry of the weight space itself.
Rather than adjusting layer-wise learning rates manually, the authors of Scalable Optimization in the Modular Norm (Tim Large et al.) introduce a recursively defined "modular norm" for neural network weights. By normalizing the updates of standard optimizers like Adam or SGD relative to this norm, the optimal learning rate becomes naturally invariant to both network width and depth.
Similarly, Thomas Pethick et al. propose SCION in Training Deep Learning Models with Norm-Constrained LMOs, which utilizes norm-constrained Linear Minimization Oracles to adapt to a network's geometric structure a priori, facilitating zero-shot transfer across widths while reducing memory overhead.
Other researchers focus on modifying the optimizer's update rules directly. Katie E Everett et al. show in Scaling Exponents Across Parameterizations and Optimizers that standard parameterizations can achieve reliable hyperparameter transfer if certain feature-alignment assumptions are relaxed. They introduce Adam-atan2, a scale-invariant variant of the Adam optimizer that eliminates the sensitive $\epsilon$ hyperparameter entirely, stabilizing optimization across scales.
Transferring Across Tokens, Batches, and Graphs
Model width is not the only dimension that alters optimal hyperparameters. As training scales up, practitioners simultaneously scale the batch size, the total number of training tokens, or the structural size of the input data.
Token Horizons
The optimal learning rate is not a static property of an architecture; it depends heavily on how long the model is trained. In Scaling Optimal LR Across Token Horizons, Johan Bjorck et al. reveal that the optimal learning rate decreases predictably as the total token horizon increases. This relationship allows practitioners to run short-duration proxy runs to find the optimal learning rate and scale it up to trillions of tokens via a zero-overhead mathematical rule.
Batch Sizes
Altering the batch size changes the gradient noise variance, which typically breaks standard learning rate schedules. To decouple these variables, Laurence Aitchison et al. develop a batch-size invariant version of Adam in Batch size invariant Adam. In a similar vein, Yikang Shen et al. introduce the Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler, which leverages a power-law relationship to make learning rate trajectories agnostic to both batch size and token count, enabling early stopping and continual training without performance degradation.
Graph Subsets
In relational and structural domains like graph machine learning, scaling challenges manifest in the size of the graph dataset. The authors of Efficient Hyper-parameter Search for Knowledge Graph Embedding (Yongqi Zhang et al.) demonstrate that hyperparameter performance rankings on small, sampled subgraphs correlate strongly with their rankings on the full knowledge graph. This high correlation enables a two-stage optimization strategy where configurations are isolated cheaply on subgraphs before being deployed at full scale.
Limits and Practical Debugging
While cross-scale transfer frameworks drastically reduce the search space for large models, certain hyperparameters remain challenging to transfer zero-shot. As noted by Ge Yang et al., regularization parameters like dropout or weight decay depend on the ratio between a model's capacity and the dataset size. When a model scales up while the dataset remains fixed, this ratio shifts, meaning that optimal regularization settings must still be adjusted.
However, even when perfect transfer is elusive, proxy models can still be used to diagnose large-scale failures. In Small-scale proxies for large-scale Transformer training instabilities, Mitchell Wortsman et al. demonstrate that catastrophic training instabilities—such as loss spikes that typically plague multi-billion parameter models—can be intentionally reproduced in small proxy models by "stress-testing" them with high learning rates. By monitoring internal metrics like activation and gradient norms on these cheap proxies, researchers can predict and mitigate training failures before deploying massive production runs.