On the Parameterization of Second-Order Optimization Effective towards the Infinite Width

Satoki Ishikawa, Ryo Karakida

Training massive neural networks presents a significant logistical challenge: hyperparameters that are optimal for a small model often fail completely when applied to a larger one. For first-order optimizers like SGD and Adam, the "Maximal Update Parameterization" ($\mu$P) framework solved this by enabling zero-shot hyperparameter transfer across model scales. However, second-order methods—which use curvature information to accelerate convergence—have largely been left behind.

This paper bridges that gap by extending the $\mu$P framework to second-order optimization methods, specifically K-FAC and Shampoo. By identifying the correct scaling for initialization, learning rates, and damping terms, the authors enable stable feature learning and seamless hyperparameter transfer as neural networks grow wider.

The Challenge of Second-Order Scaling

Second-order optimizers introduce additional hyperparameters, most notably the damping term ($\rho$), which is crucial for preventing numerical instability. Tuning these extra terms as models scale is notoriously difficult.

Without proper scaling, wider networks often fall into the "lazy regime." In this state, the network acts essentially like a linear model, failing to learn complex, hierarchical features effectively. The primary goal of this research is to find the precise scaling rules for second-order hyperparameters that keep feature learning stable, regardless of how wide the network becomes.

Extending $\mu$P to Second-Order Methods

The authors analyze the infinite-width limit of neural networks using a "one-step update" approach. They derive a specific version of the abc-parameterization, identifying the correct scaling exponents for weight matrices, initialization variance, and learning rates relative to the network width $M$.

Key methodological contributions include:

  • Deriving $\mu$P for K-FAC and Shampoo: The authors identify scaling laws that ensure parameter updates remain $\Theta(1)$. This prevents updates from vanishing or exploding as the network width increases.
  • Rescaled Damping: A critical discovery is that standard damping heuristics (like those typically used in K-FAC) do not scale correctly under $\mu$P. To fix this, the authors propose a new trace-based rescaled damping that maintains the validity of the second-order update across all scales.
  • Analytical Framework: By utilizing the "push-through identity," the authors evaluate the order of parameter updates, revealing exactly how preconditioning matrices interact with network width.
Figure 1: $\mu$P achieves stable feature learning ($\Delta h$) across different widths, whereas standard parameterization (SP) shows width-dependent decay or growth.

Key Findings and Results

The theoretical framework was validated across standard vision (CIFAR, ImageNet) and language (WikiText2) benchmarks using various architectures, including MLPs, CNNs, and ResNets.

  • K-FAC's Natural Scaling: A surprising insight is that K-FAC with $\mu$P operates effectively with a constant learning rate regardless of width. While SGD requires the learning rate to be scaled by $1/M$ for the output layer, K-FAC’s preconditioning naturally absorbs this necessary scaling.
  • Zero-Shot Hyperparameter Transfer: The most practical benefit of this parameterization is the ability to transfer hyperparameters. For models using $\mu$P, the optimal learning rate found on a small model remains optimal for a significantly larger model, saving massive amounts of compute.
  • Avoiding the Lazy Regime: The proposed parameterization ensures that hidden layers continue to evolve and learn features at the maximum possible scale, rather than remaining stuck near their initial values.
Figure 4: Under $\mu$P, test loss trajectories remain consistent across different network widths, enabling reliable hyperparameter transfer.

Limitations and Future Work

While this study provides a robust framework for scaling width, scaling with respect to depth remains a separate and complex challenge for future research. Additionally, the empirical evaluations focused on MLPs, CNNs, and ResNets; the authors note that finite-width effects in Transformers might be non-negligible and require further dedicated study. Finally, while empirical results strongly support the theory throughout training, the formal derivation relies on a one-step update analysis, leaving a rigorous multi-step proof as an open problem.

Summary updated

Abstract

Second-order optimization has been developed to accelerate the training of deep neural networks and it is being applied to increasingly larger-scale models. In this study, towards training on further larger scales, we identify a specific parameterization for second-order optimization that promotes feature learning in a stable manner even if the network width increases significantly. Inspired by a maximal update parametrization, we consider a one-step update of the gradient and reveal the appropriate scales of hyperparameters including random initialization, learning rates, and damping terms. Our approach covers two major second-order optimization algorithms, K-FAC and Shampoo, and we demonstrate that our parametrization achieves higher generalization performance in feature learning. In particular, it enables us to transfer the hyperparameters across models with different widths.
BibTeX
@inproceedings{ishikawa2024parameterization,
  author = {Satoki Ishikawa and Ryo Karakida},
  title = {{On the Parameterization of Second-Order Optimization Effective towards the Infinite Width}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=g8sGBSQjYk},
}
Figure