Abstract
Batch Normalization (BN) is a ubiquitous technique in deep learning, yet its theoretical convergence properties have remained largely unproven due to the non-gradient updates of its normalization parameters. This paper introduces Diminishing Batch Normalization (DBN), a generalization of BN that employs a diminishing moving average for updating mean and variance statistics. We provide the first theoretical convergence guarantee for a BN-like algorithm in non-convex settings, proving that DBN reaches a stationary point. Empirical results on MNIST, Network Intrusion, and CIFAR-10 demonstrate that DBN not only provides theoretical stability but also outperforms standard BN in practice.
Introduction: The "Black Box" of Batch Normalization
Batch Normalization has become a standard component in modern neural networks, celebrated for its ability to accelerate training and reduce internal covariate shift. However, its theoretical foundation is surprisingly sparse. The primary challenge in proving BN's convergence lies in its hybrid update logic: while weights are updated via gradient descent, the normalization parameters (mean and variance) are typically recalculated based on mini-batch statistics. This departure from pure gradient-based optimization invalidates classical convergence proofs.
The Diminishing Batch Normalization (DBN) Algorithm
To bridge this theoretical gap, we propose the DBN algorithm. DBN modifies the standard BN update by incorporating a weighted moving average of past statistics.
The core innovation is the parameter $\alpha^{(m)}$, which controls the influence of the current batch's statistics. When $\alpha^{(m)} = 1$, the algorithm reverts to standard BN. By setting $\alpha$ to a diminishing sequence (e.g., $1/m$), the normalization layers become increasingly stable as training progresses, reflecting the distribution of the entire dataset rather than just the current mini-batch.
Theoretical Framework and Convergence
We analyze a two-layer neural network model with an arbitrary activation function and a non-convex objective. The architecture includes a hidden layer followed by a BN layer before the output.
Our analysis proves that if the step sizes ($\eta$) and the diminishing weights ($\alpha$) satisfy specific summability conditions (square-summable but not summable), the algorithm is guaranteed to converge to a stationary point. This represents the first formal convergence proof for an algorithm incorporating BN layers in a non-convex setting.
Experimental Evaluation
We evaluated DBN against standard BN across three benchmarks:
- MNIST: Using a four-layer fully connected network (FNN), DBN ($\alpha=0.5$) achieved a test error of 1.84%, compared to 2.70% for standard BN.
- Network Intrusion (NI): On the KDD 1999 dataset, DBN ($\alpha=0.25$) reached an error of 7.24%, outperforming BN's 7.69%.
- CIFAR-10: In a complex CNN, DBN ($\alpha=0.001$) achieved a test error of 16.28%, a significant improvement over BN's 17.31%.
A key finding was the robustness of $\alpha = 0.25$, which consistently ranked among the top performers across all datasets. Conversely, setting $\alpha=0$ (no updates) led to catastrophic failure, emphasizing that while diminishing updates are beneficial, continuous adaptation to the data distribution is essential.
Conclusion and Future Work
DBN provides a mathematically grounded alternative to standard Batch Normalization that offers both theoretical guarantees and improved empirical performance. By stabilizing the normalization statistics through historical averages, DBN smooths the optimization landscape.
Future work will focus on extending the theoretical proof to deeper networks and stochastic gradient settings. While the current analysis assumes a deterministic "full gradient" approach to maintain mathematical tractability, the empirical success of DBN with stochastic gradients suggests that a broader proof is within reach.
Summary updated