Abstract
This paper introduces Natural Langevin Dynamics, a method that integrates Bayesian posterior sampling with the geometric efficiency of natural gradients. By employing a "quasi-diagonal" approximation of the Fisher Information Matrix (FIM), the authors make preconditioned Stochastic Gradient Langevin Dynamics (SGLD) computationally feasible for deep neural networks. The proposed method achieves regularization performance on par with established techniques like Dropout, while providing a theoretically grounded approach to noise injection in parameter space.
Introduction
Deep neural networks are prone to overfitting, often finding point estimates that do not generalize well. Bayesian methods offer a solution by considering the entire posterior distribution of weights. Stochastic Gradient Langevin Dynamics (SGLD) is a popular technique for sampling from this posterior by adding Gaussian noise to standard Stochastic Gradient Descent (SGD).
However, standard SGLD treats the parameter space as Euclidean, ignoring the complex curvature of neural network loss landscapes. In reality, some parameters are significantly more sensitive than others. Ideally, one would "precondition" the noise using the Fisher Information Matrix (FIM), which describes the local geometry. However, for modern networks, the FIM is far too massive to compute or invert directly. This paper addresses this bottleneck by proposing a computationally efficient approximation.
Methods
The authors propose a middle ground between simple diagonal approximations and intractable full-matrix calculations through Natural Langevin Dynamics.
- Preconditioning: The inverse FIM acts as a preconditioner for both the gradient step and the injected noise. This aligns the noise with the actual shape of the Bayesian posterior, improving sampling efficiency.
- Quasi-Diagonal Approximation: This is the core technical contribution. Instead of a full matrix, the authors use a sparse structure that stores the diagonal and the first row/column for blocks of parameters (typically grouped by neuron). This captures essential correlations without the quadratic computational cost.
- Invariance Properties: Unlike standard diagonal approximations (like those used in RMSProp), this quasi-diagonal approach maintains "invariance." The algorithm performs consistently even if inputs are rescaled or activation functions are swapped (e.g., from sigmoid to tanh).
- Ensembling: To make predictions, the method averages the outputs of several different weight sets sampled along the SGLD trajectory, rather than relying on a single point estimate.
Results and Discussion
The approach was evaluated on the MNIST dataset using a 400-400 hidden layer ReLU network. The experiments compared various preconditioning strategies—Euclidean, RMSProp, and Quasi-Diagonal—against standard SGD and Dropout baselines.
As shown in the results above, the Quasi-Diagonal Outer Product (QDOP) with ensembling achieved a test accuracy of 98.38%. This outperformed standard Euclidean SGLD (98.10%) and RMSProp-preconditioned SGLD (98.17%). While it remains slightly behind Dropout (98.61%), it demonstrates that Bayesian sampling can be a viable alternative to heuristic regularization.
A key insight from the data is the stark difference between using the "Posterior Mean" (averaging the weights) and "Ensembling" (averaging the outputs). Ensembling was significantly more accurate, confirming that the diversity of the sampled models is where the Bayesian strength lies. Using a single posterior mean model often performed worse than standard SGD.
Conclusion
Natural Langevin Dynamics with quasi-diagonal approximations offers a "sweet spot" in the trade-off between computational efficiency and geometric accuracy. It captures enough parameter correlation to improve sampling significantly over diagonal methods without the computational explosion of a full Fisher matrix. While current experiments were limited to MNIST, the efficiency of the method suggests it could scale to larger architectures. Future work will focus on applying these approximations to more complex models and larger-scale vision tasks.
Summary updated