Training massive neural networks presents a significant logistical challenge: hyperparameters that are optimal for a small model often fail completely when applied to a larger one. For first-order optimizers like SGD and Adam, the "Maximal Update Parameterization" ($\mu$P) framework solved this by enabling zero-shot hyperparameter transfer across model scales. However, second-order methods—which use curvature information to accelerate convergence—have largely been left behind.
This paper bridges that gap by extending the $\mu$P framework to second-order optimization methods, specifically K-FAC and Shampoo. By identifying the correct scaling for initialization, learning rates, and damping terms, the authors enable stable feature learning and seamless hyperparameter transfer as neural networks grow wider.
The Challenge of Second-Order Scaling
Second-order optimizers introduce additional hyperparameters, most notably the damping term ($\rho$), which is crucial for preventing numerical instability. Tuning these extra terms as models scale is notoriously difficult.
Without proper scaling, wider networks often fall into the "lazy regime." In this state, the network acts essentially like a linear model, failing to learn complex, hierarchical features effectively. The primary goal of this research is to find the precise scaling rules for second-order hyperparameters that keep feature learning stable, regardless of how wide the network becomes.
Extending $\mu$P to Second-Order Methods
The authors analyze the infinite-width limit of neural networks using a "one-step update" approach. They derive a specific version of the abc-parameterization, identifying the correct scaling exponents for weight matrices, initialization variance, and learning rates relative to the network width $M$.
Key methodological contributions include:
- Deriving $\mu$P for K-FAC and Shampoo: The authors identify scaling laws that ensure parameter updates remain $\Theta(1)$. This prevents updates from vanishing or exploding as the network width increases.
- Rescaled Damping: A critical discovery is that standard damping heuristics (like those typically used in K-FAC) do not scale correctly under $\mu$P. To fix this, the authors propose a new trace-based rescaled damping that maintains the validity of the second-order update across all scales.
- Analytical Framework: By utilizing the "push-through identity," the authors evaluate the order of parameter updates, revealing exactly how preconditioning matrices interact with network width.
Key Findings and Results
The theoretical framework was validated across standard vision (CIFAR, ImageNet) and language (WikiText2) benchmarks using various architectures, including MLPs, CNNs, and ResNets.
- K-FAC's Natural Scaling: A surprising insight is that K-FAC with $\mu$P operates effectively with a constant learning rate regardless of width. While SGD requires the learning rate to be scaled by $1/M$ for the output layer, K-FAC’s preconditioning naturally absorbs this necessary scaling.
- Zero-Shot Hyperparameter Transfer: The most practical benefit of this parameterization is the ability to transfer hyperparameters. For models using $\mu$P, the optimal learning rate found on a small model remains optimal for a significantly larger model, saving massive amounts of compute.
- Avoiding the Lazy Regime: The proposed parameterization ensures that hidden layers continue to evolve and learn features at the maximum possible scale, rather than remaining stuck near their initial values.
Limitations and Future Work
While this study provides a robust framework for scaling width, scaling with respect to depth remains a separate and complex challenge for future research. Additionally, the empirical evaluations focused on MLPs, CNNs, and ResNets; the authors note that finite-width effects in Transformers might be non-negligible and require further dedicated study. Finally, while empirical results strongly support the theory throughout training, the formal derivation relies on a one-step update analysis, leaving a rigorous multi-step proof as an open problem.
Summary updated