Abstract
Second-order optimization has been developed to accelerate the training of deep neural networks and it is being applied to increasingly larger-scale models. In this study, towards training on further larger scales, we identify a specific parameterization for second-order optimization that promotes feature learning in a stable manner even if the network width increases significantly. Inspired by a maximal update parameterization, we consider a one-step update of the gradient and reveal the appropriate scales of hyperparameters including random initialization, learning rates, and damping terms. Our approach covers two major second-order optimization algorithms, K-FAC and Shampoo, and we demonstrate that our parameterization achieves higher generalization performance in feature learning. In particular, it enables us to transfer the hyperparameters across models with different widths.
Introduction
Training massive neural networks is often a high-stakes game of trial and error, largely due to the complexity of hyperparameter (HP) tuning. While "Maximal Update Parameterization" ($\mu$P) has recently revolutionized first-order training (like SGD and Adam) by allowing HPs to transfer from small proxy models to giant ones, second-order optimizers have historically lacked a similar theoretical framework.
As neural networks grow wider, their training dynamics change. Standard parameterizations (SP) often lead to "lazy learning," where the model fails to learn complex features. This study extends the $\mu$P framework to second-order optimization (SOO) methods like K-FAC and Shampoo, providing a theoretical recipe for scaling that ensures stable feature learning at any width.
Methodology
The authors analyze the "infinite width limit" of neural networks to derive a specific version of $\mu$P for SOO. They focus on two primary algorithms: K-FAC (Kronecker-factored Approximate Curvature) and Shampoo. By utilizing the "ABC-parameterization" framework, they identified the exact scaling exponents for weights, initializations, and learning rates.
A critical innovation is the introduction of "rescaled damping." The damping term, which stabilizes SOO, typically requires manual tuning as models scale. The authors' proposed rescaling ensures that the optimal damping value remains width-independent, preventing the feature learning decay often seen in standard heuristics.
Key Insights and Ideas
- K-FAC's Self-Scaling: A surprising finding is that K-FAC's preconditioning naturally handles much of the scaling needed for $\mu$P. Unlike SGD, which requires scaling down the learning rate as width increases, K-FAC can often utilize a constant learning rate across different model sizes.
- Shampoo Requirements: Shampoo behaves more like first-order methods and requires explicit learning rate scaling (specifically $1/\sqrt{M}$ for hidden layers) to maintain stable feature learning.
- Implicit Bias at Zero Init: The study discovered that K-FAC has a unique implicit bias: if the last layer is initialized to zero, K-FAC can collapse the model into a kernel solution (NNGP) in a single step, which can hinder feature learning if not properly managed.
Experimental Results
The authors validated their theory across diverse tasks (MNIST, CIFAR, ImageNet, WikiText2) and architectures (MLPs, CNNs, ResNets).
Zero-Shot HP Transfer
The proposed $\mu$P enabled successful zero-shot HP transfer. For instance, the optimal learning rate and damping term found on a model with width 128 remained optimal for a model with width 16,384. This significantly reduces the compute cost for tuning large-scale models.
Generalization and Stability
In language tasks using Word2Vec, $\mu$P prevented the performance collapse observed in SP at high widths. While SP performance dropped to zero as width increased, $\mu$P remained stable and even improved with width.
Conclusion
This work provides a robust theoretical foundation for scaling second-order optimizers. By enabling stable feature learning and hyperparameter transferability, it makes SOO a viable choice for the next generation of massive-scale models. Future research is needed to extend these principles to depth scaling and Transformer-based architectures, which are central to current large language model development.
Summary updated