Some Fundamental Aspects about Lipschitz Continuity of Neural Networks

Grigory Khromov, Sidak Pal SinghICLR 2024 · OpenReview

Lipschitz continuity is a fundamental mathematical property that dictates how much a model's output changes in response to a small change in its input. In deep learning, a low Lipschitz constant implies a "smooth" function, which is generally associated with better generalization and resistance to adversarial attacks.

However, calculating the true Lipschitz constant of a deep neural network is an NP-hard problem. Consequently, most research has focused on either enforcing artificial Lipschitz constraints—which can degrade model accuracy—or finding tighter mathematical upper bounds. This paper takes a refreshingly empirical path: it investigates how the Lipschitz constant naturally evolves in standard, unconstrained networks during training, bridging the gap between theoretical smoothness and the practical reality of large-scale models like ResNets and Vision Transformers (ViTs).

The "Sandwiching" Strategy

To track the Lipschitz constant ($C$) without an exact calculation, the authors employ a bounding strategy:

  1. Lower Bound ($C_{lower}$): Calculated using the maximum spectral norm of the Jacobian matrix across a set of input samples. This serves as a measure of "effective" local smoothness.
  2. Upper Bound ($C_{upper}$): Computed using the standard method of multiplying the spectral norms of each individual layer's weight matrix.

By tracking these bounds across various architectures (FCNs, CNNs, ResNets, ViTs) and datasets (from toy 2D clusters to ImageNet), the authors characterize the true phenomenology of neural network smoothness.

Key Insights

The Laxity of Theoretical Upper Bounds

One of the most striking findings is the sheer uselessness of standard theoretical upper bounds for deep networks. For deep models like ResNet50, the theoretical upper bound can skyrocket to astronomical figures (as high as $10^{40}$), while the empirical lower bound remains vastly smaller, sitting around $10^3$.

Comparison of upper and lower Lipschitz bounds over epochs, highlighting the massive divergence where upper bounds reach astronomical values.

This massive discrepancy suggests that relying on standard upper bounds provides almost no meaningful insight into the actual functional behavior or smoothness of deep networks in practice.

The Fidelity of Lower Bounds and Training Growth

Conversely, the lower bound ($C_{lower}$) proves to be a surprisingly faithful proxy for the true Lipschitz constant. In toy examples where the true constant can be estimated via dense sampling, $C_{lower}$ closely tracks the true value.

Furthermore, tracking these bounds reveals that networks do not remain near their initialization state (contrary to some "lazy training" theories). Instead, the Lipschitz constant grows significantly—often by 40x to 60x—as the network learns features and fits the training data.

Tracking Lipschitz bounds during training shows that the lower bound closely tracks the true estimated constant, while the upper bound diverges.

Lipschitz Double Descent

Just as test error often exhibits a "Double Descent" phenomenon—peaking at the interpolation threshold before dropping as model capacity increases—the Lipschitz constant follows a remarkably similar trend. Experiments on CIFAR-100 demonstrate that the peak in the Lipschitz constant aligns almost perfectly with the peak in test loss at the interpolation threshold. As models become massively over-parameterized beyond their ability to merely fit the training set, they actually become smoother.

The Smoothness of Memorization

The study also sheds light on how models handle noisy labels. When trained with label noise, the Lipschitz constant naturally spikes because the model must become highly "wiggly" to memorize random noise. However, massively over-parameterized models are able to fit this noise more smoothly than smaller models. This provides a compelling functional explanation for why larger models are generally more robust to label noise.

Scaling Up and Future Directions

The authors successfully scaled their lower-bound estimation to ImageNet, demonstrating that even in complex, real-world settings, the Lipschitz constant eventually saturates and stabilizes rather than growing indefinitely.

While calculating Jacobians for every sample is computationally expensive (requiring subsampling strategies on large datasets like ImageNet), the statistical evidence suggests these estimates remain representative. Future work aims to explore whether these smoothness trends hold in Large Language Models (LLMs) or Reinforcement Learning environments, and to develop a rigorous, closed-form mathematical proof for the observed "Lipschitz Double Descent."

Summary updated

Abstract

Lipschitz continuity is a crucial functional property of any predictive model, that naturally governs its robustness, generalisation, as well as adversarial vulnerability. Contrary to other works that focus on obtaining tighter bounds and developing different practical strategies to enforce certain Lipschitz properties, we aim to thoroughly examine and characterise the Lipschitz behaviour of Neural Networks. Thus, we carry out an empirical investigation in a range of different settings (namely, architectures, datasets, label noise, and more) by exhausting the limits of the simplest and the most general lower and upper bounds. As a highlight of this investigation, we showcase a remarkable fidelity of the lower Lipschitz bound, identify a striking Double Descent trend in both upper and lower bounds to the Lipschitz and explain the intriguing effects of label noise on function smoothness and generalisation.
BibTeX
@inproceedings{khromov2024some,
  author = {Grigory Khromov and Sidak Pal Singh},
  title = {{Some Fundamental Aspects about Lipschitz Continuity of Neural Networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=5jWsW08zUh},
}
Figure