On the distance between two neural networks and the stability of learning

Jeremy Bernstein, Arash Vahdat, Yisong Yue, Ming-Yu Liu

TL;DR

Researchers from Caltech and NVIDIA have developed a new optimization algorithm called Fromage (Frobenius matched gradient descent) that significantly reduces the need for learning rate tuning. By replacing the traditional "quadratic trust region" with a novel distance metric called deep relative trust, they created a learner that can train everything from 50-layer deep networks to GANs and Transformers using a single, stable default learning rate.

The Problem: The Fragility of the Learning Rate

In deep learning, the learning rate ($\eta$) is arguably the most frustrating hyperparameter to tune. If it’s too high, the model explodes; if it’s too low, training takes forever. This happens because standard optimization theory assumes a "quadratic trust region"—the idea that the loss surface behaves like a simple bowl locally.

However, neural networks are compositional (built layer upon layer). A small change in the first layer's weights can have an exponentially large impact on the final output. Because standard optimizers like SGD and Adam don't account for this "deep" structure, practitioners spend countless GPU hours performing grid searches just to find a learning rate that doesn't crash the training.

Methods: Deep Relative Trust and Fromage

To address this, the authors introduce two major contributions:

  1. Deep Relative Trust: Through perturbation analysis, they prove that the distance between two neural networks isn't best measured by Euclidean distance (how much the parameters changed in total), but by a product of relative changes across layers. They show that in deep networks, the gradient breaks down "quasi-exponentially" relative to the size of the update.
  2. Fromage (Frobenius matched gradient descent): Based on this new theory, they derive a learning rule that constrains the relative change of each layer. While similar to the LARS optimizer, Fromage includes a crucial mathematical correction: a prefactor of $1/\sqrt{1+\eta^2}$.

This prefactor prevents "compounding growth," a phenomenon where weight norms grow uncontrollably in layers that are invariant to rescaling (like those using Batch Norm).

Algorithm 1 (Fromage) and a comparison showing stable layerwise parameter norms versus LARS.

Key Insights

  • The "Quasi-Exponential" Wall: As networks get deeper, the "trust region" (the area where the gradient is actually useful) shrinks dramatically. This explains why deep networks without skip connections are notoriously hard to train with standard SGD.
  • Relative Size Matters: The most effective way to stabilize training is to ensure that the update to a layer's weights is proportional to the current "size" (Frobenius norm) of those weights.
  • The Geometry of Invariance: Many modern layers (like those with Spectral Norm or Batch Norm) are "rescaling invariant." Standard updates always increase the weight norm in these layers. Fromage’s specific mathematical structure explicitly corrects for this, preventing numerical overflow.

Key Results and Benchmarks

The authors validated Fromage across a diverse set of standard benchmarks, including MNIST, CIFAR-10, ImageNet, GANs, and NLP tasks (Wikitext-2).

  • Depth Defiance: While off-the-shelf optimizers like SGD and Adam failed to train simple multi-layer perceptrons (MLPs) deeper than 25 layers, Fromage successfully trained them up to 50 layers deep without any special tricks like skip connections or Batch Norm.
Training accuracy across different network depths, showing Fromage succeeding up to 50 layers where SGD and Adam fail.
  • The "Universal" 0.01: In a striking result, the authors used a single learning rate ($\eta = 0.01$) for ImageNet, CIFAR-10, GANs, and Transformers. In almost every case, Fromage with this default setting outperformed or matched carefully tuned versions of SGD and Adam.
  • Stability in GANs: GANs are notoriously unstable. Fromage achieved a significantly better FID score (16 vs. 23.7 for Adam) on CIFAR-10, suggesting the "deep relative trust" model is particularly well-suited for competitive min-max optimization.

Limitations and Future Work

  • Overfitting: Fromage can overfit the training set more aggressively than other optimizers. To counter this, the authors implemented a constraint ensuring layer norms didn't exceed their initial values.
  • Skip Connections: Most modern architectures (like ResNets) use skip connections, which the authors hypothesize reduce the "effective depth" of the network. Future work is needed to see how deep relative trust interacts with these architectural shortcuts.
  • Theoretical Assumptions: The current proof assumes weight matrices are full rank and that nonlinearities "transmit" a certain fraction of their input (like Leaky ReLU). While it works in practice for standard ReLU, the formal theory for sparse activations is still being refined.

Summary updated

Abstract

This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions. The analysis leads to a new distance function called deep relative trust and a descent lemma for neural networks. Since the resulting learning rule seems to require little to no learning rate tuning, it may unlock a simpler workflow for training deeper and more complex neural networks. The Python code used in this paper is here: https://github.com/jxbz/fromage.
BibTeX
@inproceedings{bernstein2020distance,
  author = {Jeremy Bernstein and Arash Vahdat and Yisong Yue and Ming-Yu Liu},
  title = {{On the distance between two neural networks and the stability of learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=4qawIoEJDy},
}
Figure