Abstract
The multiplicative structure of parameters and input data in the first layer of neural networks is explored to build connection between the landscape of the loss function with respect to parameters and the landscape of the model function with respect to input data. By this connection, it is shown that flat minima regularize the gradient of the model function, which explains the good generalization performance of flat minima. Then, we go beyond the flatness and consider high-order moments of the gradient noise, and show that Stochastic Gradient Descent (SGD) tends to impose constraints on these moments by a linear stability analysis of SGD around global minima. Together with the multiplicative structure, we identify the Sobolev regularization effect of SGD, i.e. SGD regularizes the Sobolev seminorms of the model function with respect to the input data. Finally, bounds for generalization error and adversarial robustness are provided for solutions found by SGD under assumptions of the data distribution.
Introduction
Deep learning practitioners have long observed that Stochastic Gradient Descent (SGD) is not just an optimization tool, but a source of "implicit regularization." Models trained with SGD tend to find "flat minima" in the loss landscape, which generalize significantly better than the "sharp minima" often found by full-batch Gradient Descent (GD).
However, two major questions remained:
- Why does SGD specifically seek out these flat minima?
- Why do flat minima actually lead to better generalization on unseen data?
Existing theories often rely on Stochastic Differential Equations (SDEs) to model SGD noise, but these approximations only hold when learning rates are very small. This paper seeks a more robust explanation that works for practical learning rates and connects the geometry of the loss landscape (how the error changes with parameters) to the geometry of the model function (how the output changes with input data).
Methods
The authors employ two primary theoretical pillars:
- Linear Stability Analysis: Instead of treating SGD as a continuous process (SDE), they analyze it as a discrete dynamical system. They define "$k$-th order linear stability," which looks at whether the moments of the SGD iterator remain bounded near a minimum. They prove that for a minimum to be stable under SGD, the gradients of the loss must satisfy specific constraints.
- Multiplicative Structure Exploitation: The authors point out a simple but powerful fact: in the first layer of almost any neural network, parameters ($W_1$) and input data ($x$) are multiplied together ($W_1x$). Because of this, a "flat" landscape in the parameter space (where the derivative with respect to $W$ is small) mathematically forces the model to be "smooth" in the input space (where the derivative with respect to $x$ is small).
By combining these, they show that SGD's requirement for stability acts as a Sobolev regularizer, effectively penalizing high-frequency oscillations in the model function relative to the input data. The theoretical findings were validated using standard benchmarks including FashionMNIST, CIFAR10, and CIFAR100.
Results and Key Insights
The core of the paper's contribution is the "Input-Smoothness" connection. Flatness in the loss landscape is not just an abstract geometric property; it serves as a proxy for Lipschitz continuity and Sobolev smoothness in the input space. If the loss is insensitive to moving the weights, the model output is likely insensitive to small changes in the input.
Empirical Correlation and Hyperparameter Impact
Experiments confirm a near-perfect correlation between the norm of the gradients with respect to weights ($g_W$) and the norm of the gradients with respect to inputs ($g_x$). This confirms that minimizing one (which SGD does implicitly) effectively minimizes the other.
As shown in the figure above, a larger learning rate ($\eta$) and smaller batch size ($B$) strengthen this Sobolev regularization effect, pushing the model towards regions with smaller gradient norms. This provides a theoretical explanation for why these hyperparameter settings often lead to better-performing models.
Beyond Sharpness
While previous work focused on "sharpness" (the 2nd moment of noise), this paper generalizes the theory to higher-order moments ($k$-th order stability). This provides a much more granular characterization of the regularization effect. The authors argue that linear stability is a better predictor of SGD behavior than diffusion theory. In linear stability, the escape time from a "bad" (sharp) minimum depends on the logarithm of the barrier height, aligning better with empirical observations than the exponential dependency suggested by SDE models.
Generalization and Robustness
The authors derived a formal bound for generalization error: $Generalization\ Error \lesssim n^{-\frac{1}{2k}}\delta^2 + \epsilon M^2$. This shows that as the stability order ($k$) increases or the data distribution becomes more concentrated on low-dimensional manifolds, the error decreases. Furthermore, because SGD forces the model to be smooth with respect to inputs, it provides an implicit adversarial defense. A smooth function is naturally harder to "break" with the tiny, high-frequency perturbations used in adversarial attacks.
Discussion and Future Work
While the results are compelling, there are limitations. The generalization bounds are most effective when the input data lies on a low-dimensional manifold; they might weaken for extremely high-dimensional, "noisy" data. Additionally, the derived bounds depend on the norm of the first-layer weights ($W_1$). In networks with ReLU activations, these norms can technically grow very large without changing the model function due to scaling invariance. While empirical evidence shows SGD keeps these layers "balanced," a more rigorous theoretical treatment of this invariance is needed. Finally, the current analysis focuses heavily on the multiplicative structure of the first layer. Extending this explicitly to the multiplicative interactions in deeper layers (like those in Transformers or ResNets) is a promising path for future research.
Summary updated