Abstract: The choice of batch-size in a stochastic optimization algorithm plays a substantial role for both optimization and generalization. Increasing the batch-size used typically improves optimization but often leads to a "generalization gap," where the model performs worse on unseen data compared to small-batch training. This study investigates injecting structured noise—specifically based on the diagonal of the Fisher Information Matrix—into large-batch gradients to bridge this gap while maintaining computational efficiency.
Introduction
In stochastic gradient descent (SGD), mini-batch size presents a fundamental trade-off. Large batches enable better gradient estimation and superior hardware parallelization, yet they frequently converge to "sharp" minima that generalize poorly. Small batches, conversely, possess intrinsic noise that acts as a regularizer, steering the optimization toward "flatter" regions of the loss landscape associated with better generalization. The central challenge is achieving the hardware efficiency of large-batch training without sacrificing the final model accuracy.
Methodology
The authors propose a modified large-batch SGD update rule that injects Gaussian noise scaled by a specific covariance matrix to mimic the beneficial noise of small-batch training. While adding "Full Fisher" noise perfectly replicates small-batch behavior, it is computationally prohibitive. Instead, the authors propose using Diagonal Fisher noise.
At each step, the diagonal of the empirical Fisher Information Matrix is computed (requiring only one extra forward pass). Noise is then added to the gradient, where the variance for each parameter is proportional to its corresponding diagonal entry in the Fisher matrix. This approach was evaluated across standard computer vision benchmarks, including MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100, using architectures such as ResNet, VGG, and AlexNet.
Key Insights: The Importance of Noise Structure
A critical finding is that the structure of gradient noise is more important than its sheer magnitude. Different noise regimes can share the same average variance yet produce vastly different optimization trajectories.
As shown in the visualization above, structured noise aligns with the curvature of the loss landscape. For convex quadratic problems, the authors prove that convergence rates are characterized by the Frobenius norm of the noise covariance matrix. Because the diagonal Fisher matrix has a significantly smaller Frobenius norm than the full Fisher matrix (by ignoring off-diagonal correlations), it allows for much faster convergence while still providing the necessary directional regularization to find flatter minima.
Experimental Results
The proposed method successfully closes the generalization gap. For instance, on CIFAR-10 with ResNet44, standard large-batch training (batch size 4096) achieved 91.93% accuracy, trailing the small-batch (batch size 128) performance of 93.42%. Injecting diagonal Fisher noise into the large-batch process boosted accuracy to 92.88%.
Crucially, the optimization efficiency is preserved. As illustrated in the training error curves, "Full Fisher" noise (LB fullF) significantly slows down optimization, requiring many more iterations to reach low error. In contrast, "Diagonal Fisher" noise (LB diagF) follows an optimization trajectory nearly identical to standard large-batch SGD, providing the generalization benefits of noise without the typical performance penalty.
Conclusion and Future Work
This empirical study demonstrates that structured noise, specifically aligned with the diagonal of the Fisher Information Matrix, provides a practical pathway to high-performance large-batch training. While the theoretical foundations are currently strongest for convex quadratic objectives, empirical evidence suggests these principles hold for deep non-convex networks. Future research will focus on a more rigorous theoretical bridge for deep learning and exploring "uniform stability" to further explain why diagonal structures specifically aid generalization.
Summary updated