Some Fundamental Aspects about Lipschitz Continuity of Neural Network Functions

Grigory Khromov, Sidak Pal SingharXiv 2023 · arXiv · OpenReview

Abstract

Lipschitz continuity is a simple yet crucial functional property of any predictive model for it lies at the core of the model's robustness, generalisation, as well as adversarial vulnerability. Our aim is to thoroughly investigate and characterise the Lipschitz behaviour of the functions realised by neural networks. Thus, we carry out an empirical investigation in a range of different settings (namely, architectures, losses, optimisers, label noise, and more) by exhausting the limits of the simplest and the most general lower and upper bounds. Although motivated primarily by computational hardness results, this choice nevertheless turns out to be rather resourceful and sheds light on several fundamental and intriguing traits of the Lipschitz continuity of neural network functions, which we also supplement with suitable theoretical arguments. As a highlight of this investigation, we identify a striking double descent trend in both upper and lower bounds to the Lipschitz constant with increasing network width -- which tightly aligns with the typical double descent trend in the test loss. Lastly, we touch upon the seeming (counter-intuitive) decline of the Lipschitz constant in the presence of label noise.

Introduction

In machine learning, model "smoothness" is a desirable trait. A model where small input perturbations lead to disproportionately large output changes is inherently fragile and susceptible to adversarial attacks. This smoothness is formally quantified by the Lipschitz constant, representing the maximum rate of change of the function. However, computing the exact Lipschitz constant for deep neural networks is an NP-hard problem. Most existing literature focuses on either deriving tighter theoretical bounds or enforcing smoothness through regularization. This study shifts the focus toward empirically characterizing how the Lipschitz constant behaves during standard training across various architectures and datasets.

Methodology

The researchers employ a "sandwich" approach to bound the Lipschitz constant ($C$):

  1. Lower Bound ($C_{lower}$): Defined as the maximum norm of the Jacobian matrix evaluated over the training samples. This captures the local sensitivity of the model on the data manifold.
  2. Upper Bound ($C_{upper}$): Calculated using the product of the spectral norms of each layer's weight matrix.

To validate the reliability of $C_{lower}$ as a proxy for the true constant, the authors perform a fidelity check by evaluating Jacobian norms on convex combinations of "hard" samples (those with high initial Jacobian norms). The study spans synthetic 2D data, MNIST, CIFAR-10/100, and large-scale ImageNet benchmarks using ResNet50 and Vision Transformers.

Empirical Findings

The Disparity Between Bounds

A primary observation is the extreme looseness of standard upper bounds in deep architectures. In models like ResNet50 on ImageNet, the theoretical upper bound can exceed the lower bound by over 40 orders of magnitude. This suggests that the product-of-norms approach, while mathematically sound, fails to provide a meaningful estimate of a model's actual operational smoothness.

Comparison of Lipschitz lower and upper bounds during training on ImageNet. Note the dual y-axes showing a gap of approximately 40 orders of magnitude.

Fidelity of the Lower Bound

Despite being a local measure, $C_{lower}$ proves to be a remarkably faithful indicator of the model's global sensitivity. When searching for points of higher sensitivity between training samples (via convex combinations), the resulting Jacobian norms only increase by a small factor (typically ~2x), whereas the upper bound remains trillions of times larger.

Distribution of Jacobian norms on ImageNet samples versus convex combinations of samples, demonstrating the fidelity of the lower bound.

Lipschitz Double Descent

The study identifies a "Double Descent" phenomenon in the Lipschitz constant relative to network width. As width increases, the Lipschitz constant rises until the model reaches the interpolation threshold (where it can just fit the training data). Beyond this point, in the over-parameterized regime, the Lipschitz constant decreases. This mirrors the well-known double descent in test error, suggesting that larger models are naturally biased toward smoother solutions that generalize better.

Impact of Label Noise and Training Dynamics

Training on noisy labels forces the network to learn "spiky" functions to memorize random assignments, causing the Lipschitz constant to skyrocket. Interestingly, even under these conditions, very large networks eventually find smoother ways to memorize noise compared to smaller ones. Furthermore, the Lipschitz constant is not static; it typically grows by 40x to 60x from its initial value during the training process, contradicting "lazy training" assumptions in some regimes.

Conclusion

This investigation demonstrates that simple lower bounds, specifically the maximum Jacobian norm on data, are effective tools for monitoring model smoothness. The discovery of the Lipschitz Double Descent provides a new perspective on why over-parameterized models generalize well: they are inherently smoother. Future research should explore these dynamics in non-vision domains and seek more computationally efficient methods for estimating local sensitivities in massive datasets.

Summary updated

Abstract

Lipschitz continuity is a simple yet crucial functional property of any predictive model for it lies at the core of the model's robustness, generalisation, as well as adversarial vulnerability. Our aim is to thoroughly investigate and characterise the Lipschitz behaviour of the functions realised by neural networks. Thus, we carry out an empirical investigation in a range of different settings (namely, architectures, losses, optimisers, label noise, and more) by exhausting the limits of the simplest and the most general lower and upper bounds. Although motivated primarily by computational hardness results, this choice nevertheless turns out to be rather resourceful and sheds light on several fundamental and intriguing traits of the Lipschitz continuity of neural network functions, which we also supplement with suitable theoretical arguments. As a highlight of this investigation, we identify a striking double descent trend in both upper and lower bounds to the Lipschitz constant with increasing network width -- which tightly aligns with the typical double descent trend in the test loss. Lastly, we touch upon the seeming (counter-intuitive) decline of the Lipschitz constant in the presence of label noise.
BibTeX
@misc{khromov2023some,
  author = {Grigory Khromov and Sidak Pal Singh},
  title = {{Some Fundamental Aspects about Lipschitz Continuity of Neural Network Functions}},
  year = {2023},
  eprint = {2302.10886},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.10886},
}
Figure