Abstract
This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions. The analysis leads to a new distance function called deep relative trust and a descent lemma for neural networks. Since the resulting learning rule seems to require little to no learning rate tuning, it may unlock a simpler workflow for training deeper and more complex neural networks. The Python code used in this paper is available at: https://github.com/jxbz/fromage.
Introduction
Standard optimization theory for deep learning typically relies on the assumption of Lipschitz continuous gradients, which implies a quadratic trust region. However, the compositional nature of deep neural networks causes gradients to break down quasi-exponentially as the size of the update increases. This "compositional fragility" makes training highly sensitive to the choice of learning rate ($\eta$), often requiring expensive hyperparameter grid searches to prevent gradient explosion or stalled learning.
This work introduces deep relative trust, a distance metric that accounts for the layered architecture of neural networks. By analyzing how perturbations propagate through layers, the authors derive a new descent lemma and a practical optimizer, Fromage (Frobenius matched gradient descent), designed for stable, tuning-free training.
Deep Relative Trust and the Descent Lemma
Instead of using simple Euclidean distance ($\Delta W$), the authors propose measuring the relative change in each layer. They prove that the functional change of a network is bounded by the product of these relative changes across all layers. This leads to a new descent lemma specifically for deep networks, suggesting that the optimal learning rate should be depth-dependent to maintain stability.
The Fromage Optimizer
Fromage implements these theoretical insights through a layerwise update rule that scales the gradient by the ratio of the weight norm to the gradient norm, similar to LARS. A critical addition is a rescaling prefactor, $1/\sqrt{1+\eta^2}$, which ensures that the parameter norms remain stable over time, particularly in scale-invariant layers (e.g., those with Batch Normalization).
In Fromage, the hyperparameter $\eta$ has a concrete physical meaning: it represents the percentage change in the weights of each layer per iteration. For example, setting $\eta=0.01$ corresponds to a 1% update per step.
Experimental Results
The authors evaluated Fromage across diverse tasks, including image classification (CIFAR-10, ImageNet), GAN training, and language modeling (Transformer on Wikitext-2).
The "Universal" Learning Rate
A key finding is that Fromage consistently achieves optimal or near-optimal performance with a default learning rate of $\eta = 0.01$ across all tested architectures and datasets. In contrast, standard optimizers like SGD and Adam exhibit high sensitivity, requiring task-specific tuning over several orders of magnitude to find their best settings.
Performance Highlights
- Stability at Depth: Fromage successfully trained a 50-layer MLP without skip connections or Batch Normalization, a task where SGD and Adam failed to converge.
- Optimization Efficiency: On CIFAR-10, Fromage achieved a training loss of $2.5 \times 10^{-5}$, significantly lower than tuned SGD ($1.5 \times 10^{-4}$) and Adam ($6 \times 10^{-5}$).
- GAN Training: On CIFAR-10, Fromage achieved an FID score of 16, significantly outperforming Adam (23.7) and SGD (34).
- ImageNet: Using its default settings, Fromage matched the performance of a heavily tuned SGD on ResNet-50.
Conclusion
By redefining distance in the context of deep composition, this work provides a more robust theoretical foundation for deep learning optimization. The resulting Fromage optimizer effectively eliminates the need for learning rate tuning, offering a more efficient and predictable workflow for training complex models. While challenges like overfitting in certain regimes remain, the principle of deep relative trust represents a significant step toward automated, stable deep learning.
Summary updated