On the distance between two neural networks and the stability of learning

Jeremy Bernstein, Arash Vahdat, Yisong Yue, Ming-Yu Liu

Abstract

This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions. The analysis leads to a new distance function called deep relative trust and a descent lemma for neural networks. Since the resulting learning rule seems to require little to no learning rate tuning, it may unlock a simpler workflow for training deeper and more complex neural networks. The Python code used in this paper is available at: https://github.com/jxbz/fromage.

Introduction

Standard optimization theory for deep learning typically relies on the assumption of Lipschitz continuous gradients, which implies a quadratic trust region. However, the compositional nature of deep neural networks causes gradients to break down quasi-exponentially as the size of the update increases. This "compositional fragility" makes training highly sensitive to the choice of learning rate ($\eta$), often requiring expensive hyperparameter grid searches to prevent gradient explosion or stalled learning.

This work introduces deep relative trust, a distance metric that accounts for the layered architecture of neural networks. By analyzing how perturbations propagate through layers, the authors derive a new descent lemma and a practical optimizer, Fromage (Frobenius matched gradient descent), designed for stable, tuning-free training.

Deep Relative Trust and the Descent Lemma

Instead of using simple Euclidean distance ($\Delta W$), the authors propose measuring the relative change in each layer. They prove that the functional change of a network is bounded by the product of these relative changes across all layers. This leads to a new descent lemma specifically for deep networks, suggesting that the optimal learning rate should be depth-dependent to maintain stability.

The Fromage Optimizer

Fromage implements these theoretical insights through a layerwise update rule that scales the gradient by the ratio of the weight norm to the gradient norm, similar to LARS. A critical addition is a rescaling prefactor, $1/\sqrt{1+\eta^2}$, which ensures that the parameter norms remain stable over time, particularly in scale-invariant layers (e.g., those with Batch Normalization).

Algorithm 1 for Fromage and a comparison of layerwise parameter norm stability between Fromage and LARS over 100 epochs.

In Fromage, the hyperparameter $\eta$ has a concrete physical meaning: it represents the percentage change in the weights of each layer per iteration. For example, setting $\eta=0.01$ corresponds to a 1% update per step.

Experimental Results

The authors evaluated Fromage across diverse tasks, including image classification (CIFAR-10, ImageNet), GAN training, and language modeling (Transformer on Wikitext-2).

The "Universal" Learning Rate

A key finding is that Fromage consistently achieves optimal or near-optimal performance with a default learning rate of $\eta = 0.01$ across all tested architectures and datasets. In contrast, standard optimizers like SGD and Adam exhibit high sensitivity, requiring task-specific tuning over several orders of magnitude to find their best settings.

Comparison of learning rate sensitivity for SGD, Fromage, and Adam across CIFAR-10, GAN, and Transformer tasks. Fromage shows a consistent peak at η=0.01.

Performance Highlights

  • Stability at Depth: Fromage successfully trained a 50-layer MLP without skip connections or Batch Normalization, a task where SGD and Adam failed to converge.
  • Optimization Efficiency: On CIFAR-10, Fromage achieved a training loss of $2.5 \times 10^{-5}$, significantly lower than tuned SGD ($1.5 \times 10^{-4}$) and Adam ($6 \times 10^{-5}$).
  • GAN Training: On CIFAR-10, Fromage achieved an FID score of 16, significantly outperforming Adam (23.7) and SGD (34).
  • ImageNet: Using its default settings, Fromage matched the performance of a heavily tuned SGD on ResNet-50.

Conclusion

By redefining distance in the context of deep composition, this work provides a more robust theoretical foundation for deep learning optimization. The resulting Fromage optimizer effectively eliminates the need for learning rate tuning, offering a more efficient and predictable workflow for training complex models. While challenges like overfitting in certain regimes remain, the principle of deep relative trust represents a significant step toward automated, stable deep learning.

Summary updated

Abstract

This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions. The analysis leads to a new distance function called deep relative trust and a descent lemma for neural networks. Since the resulting learning rule seems to require little to no learning rate tuning, it may unlock a simpler workflow for training deeper and more complex neural networks. The Python code used in this paper is here: https://github.com/jxbz/fromage.
BibTeX
@inproceedings{bernstein2020distance,
  author = {Jeremy Bernstein and Arash Vahdat and Yisong Yue and Ming-Yu Liu},
  title = {{On the distance between two neural networks and the stability of learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=4qawIoEJDy},
}
Figure