On Monotonic Linear Interpolation of Neural Network Parameters

James Lucas, Juhan Bae, Michael R. Zhang, Stanislav Fort, Richard S. Zemel, Roger B. GrosseICML 2021 · OpenReview

Introduction

In 2014, Ian Goodfellow and colleagues noticed a fascinating phenomenon in deep learning: despite neural network loss landscapes being high-dimensional, highly non-convex, and theoretically complex, the training loss usually decreases smoothly and monotonically along a straight line from the starting weights at initialization to the final trained weights. This property, known as Monotonic Linear Interpolation (MLI), suggested that neural network optimization might be geometrically simpler than previously assumed.

However, the landscape of deep learning has evolved significantly since 2014. Modern architectures like ResNets and Transformers, along with adaptive optimizers like Adam, have become standard. This paper investigates whether MLI is a universal truth of neural networks or a fragile byproduct of specific training setups. By combining extensive empirical testing with a novel geometric framework, the authors demonstrate that MLI is not an inherent property of network architectures and can be systematically broken by modern training techniques.

Loss landscape and interpolation showing monotonic decrease from multiple initializations.

A Geometric Framework for MLI

To move beyond qualitative observations, the authors introduce tools from differential geometry to analyze optimization paths.

While the path between initialization and the final solution is a straight line in weight space, the corresponding path in function space (the actual outputs or logits of the network) is a curve. The authors utilize a metric called Gauss length to measure how much this function-space curve "bends."

Through theoretical proofs, they establish a sufficient condition for MLI: for Mean Squared Error (MSE) loss, if the Gauss length of the path in function space is less than $\pi/2$, the network is mathematically guaranteed to satisfy the MLI property. Furthermore, using Taylor expansions, they show that "Lazy Training"—a regime where weights do not move far from their initialization—naturally leads to MLI because the function-space path remains relatively straight.

What Breaks Monotonic Linear Interpolation?

Through extensive empirical probing across various tasks (including image classification, reconstruction, and language modeling) and architectures (LeNet, VGG, ResNet, LSTMs, and Transformers), the authors discovered that MLI is surprisingly fragile. It is heavily influenced by specific optimization choices:

The "Adam" Effect

Adaptive optimizers like Adam are significantly more likely to break MLI than standard Stochastic Gradient Descent (SGD). In CIFAR-10 experiments, models trained with Adam violated MLI 3.2 times more often than those trained with SGD. While SGD tends to stay "on track," Adam often takes a more circuitous route in weight space that bypasses high-loss barriers. When you later draw a straight line between initialization and the Adam-optimized solution, that line cuts straight through the barriers, appearing as massive "bumps" in the loss interpolation.

Interpolated paths for VGG19 and ResNet18 showing Adam violating MLI with large loss bumps compared to SGD.

Large Learning Rates and Batch Normalization

High learning rates allow the optimizer to "jump" over high-loss barriers early in training, a phenomenon sometimes called the catapult phase. This movement pushes the weights far from their initialization. The authors found a strong power-law relationship between the distance moved in weight space and the average Gauss length. As networks move further from their starting point, the path in function space bends more, making MLI violations highly likely. For instance, 71% of configurations with a learning rate of 1.0 resulted in non-monotonic paths.

Similarly, adding Batch Normalization significantly increased the rate of MLI failure. Batch Norm alters the geometry of the loss landscape, allowing for larger effective optimization steps that bend the function-space path beyond the $\pi/2$ Gauss length threshold.

A Surprising Global Property

Despite its fragility to modern optimization techniques, MLI exhibits a fascinating global characteristic when it does hold. The authors found that if MLI holds for the path from the original initialization to the final solution, it often holds for paths drawn from unrelated random initializations to that same solution. This suggests that the "basin of attraction" around the optimum is quite broad and smooth in specific directions, reinforcing the idea that certain optimization trajectories are remarkably well-behaved.

Limitations and Future Directions

While this paper provides a robust geometric explanation for MLI, several open questions remain:

  • Loss Function Scope: The primary theoretical guarantee using Gauss length is derived specifically for Mean Squared Error (MSE). While empirical results show similar behavior for Cross-Entropy loss, a formal mathematical proof for Cross-Entropy remains an open challenge.
  • Optimizer Dynamics: The empirical evidence clearly shows that Adam breaks MLI, but more theoretical work is needed to understand the specific mechanics of why adaptive gradients favor paths that bend so severely in function space.
  • Mode Connectivity: The paper distinguishes between MLI (interpolation from initialization to solution) and linear mode connectivity (interpolation between two different trained solutions). Exploring the theoretical bridge between these two phenomena is a promising next step for understanding neural network loss landscapes.

Summary updated

Abstract

Linear interpolation between initial neural network parameters and converged parameters after training with stochastic gradient descent (SGD) typically leads to a monotonic decrease in the training...
BibTeX
@inproceedings{lucas2021monotonic,
  author = {James Lucas and Juhan Bae and Michael R. Zhang and Stanislav Fort and Richard S. Zemel and Roger B. Grosse},
  title = {{On Monotonic Linear Interpolation of Neural Network Parameters}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=BHlZgOY4G7q},
}
Figure