Abstract
Langevin dynamics (LD) is widely used for sampling from distributions and for optimization. In this work, we derive a closed-form expression for the expected loss of preconditioned LD near stationary points of the objective function. We use the fact that at the vicinity of such points, LD reduces to an Ornstein-Uhlenbeck process, which is amenable to convenient mathematical treatment. Our analysis reveals that when the preconditioning matrix satisfies a particular relation with respect to the noise covariance, LD's expected loss becomes proportional to the rank of the objective's Hessian. We illustrate the applicability of this result in the context of neural networks, where the Hessian rank has been shown to capture the complexity of the predictor function but is usually computationally hard to probe. Finally, we use our analysis to compare SGD-like and Adam-like preconditioners and identify the regimes under which each of them leads to a lower expected loss.
Introduction: The Challenge of the Hessian
In deep learning, the Hessian matrix—the matrix of second-order derivatives of the loss function—is a critical tool for understanding model behavior. Its eigenvalues characterize the "sharpness" of local minima, a property closely linked to generalization performance. Specifically, the rank of the Hessian indicates the dimensionality of the active parameter space, or the number of "effective" parameters the model utilizes.
However, for modern neural networks with millions or billions of parameters, the Hessian is computationally intractable. It contains trillions of entries, making direct computation, storage, or rank estimation impossible. Existing estimation techniques often rely on Hessian-vector products, which are computationally demanding and sensitive to hyperparameter tuning. This research introduces a more natural and efficient approach to probing this hidden structure by observing how the model responds to injected noise.
Theoretical Framework: From Langevin Dynamics to Hessian Rank
The authors leverage Preconditioned Langevin Dynamics (LD), essentially Gradient Descent augmented with scaled Gaussian noise. The core insight lies in the behavior of the system near a local minimum.
The OU Approximation
Near a stationary point, the loss landscape can be locally approximated as a quadratic "bowl." In this regime, Langevin Dynamics simplifies to an Ornstein–Uhlenbeck (OU) process. This mathematical framework allows for the derivation of a closed-form expression for the expected loss over time.
The Rank-Loss Relationship
The researchers discovered that by carefully balancing the noise covariance ($\Sigma$) and the preconditioner matrix ($G$) such that $\Sigma G = \sigma^2 I$, the steady-state expected loss becomes directly proportional to the rank of the Hessian. This means the value of the loss itself, under specific noise conditions, encodes the Hessian's rank. Unlike previous methods that related noise to the trace (sum of eigenvalues), this approach isolates the rank (count of non-zero eigenvalues).
Experimental Validation
The theory was validated using two primary setups:
- Linear Neural Networks: Synthetic regression tasks where the ground truth Hessian rank is known to be the product of input and output dimensions ($d_x \times d_y$).
- Denoising NN (DnCNN): A non-linear deep network trained on the MNIST dataset. This allowed for numerical verification of the Hessian rank in a real-world, albeit smaller-scale, scenario.
Key Findings
The proposed method demonstrates significant improvements in both accuracy and efficiency over existing baselines, such as the "U&S" method.
- High Precision: In linear network experiments, the method achieved an error rate as low as 0.3% for a rank of $64^2$, compared to 19.2% for the baseline.
- Spectral Coverage: On the DnCNN task, the estimation captured over 98.5% of the spectral energy of the true Hessian, confirming that the network's behavior is dominated by a small fraction of its parameters.
- Computational Efficiency: Because the method only requires standard backpropagation gradients rather than complex Hessian-vector products, it is roughly twice as fast per iteration as competing techniques.
- Stability via Preconditioning: The use of adaptive weights (similar to the Adam optimizer) as a preconditioner proved essential for maintaining stability in deep, non-linear networks, preventing the "jiggling" process from escaping the local minimum.
Conclusion and Future Directions
This work provides a novel bridge between the dynamics of noisy optimization and the geometric structure of the loss landscape. By showing that the expected loss reveals the Hessian rank, the authors offer a practical tool for measuring model complexity.
While the current theory assumes a quadratic loss landscape near stationary points, future research could explore its applicability in non-smooth regions or during the early stages of training. Furthermore, the underlying principle of using noise-induced loss to estimate the rank of symmetric matrices could find applications in various scientific fields beyond deep learning.
Summary updated