Convergence Analysis of Batch Normalization for Deep Neural Nets

Yintai Ma, Diego Klabjan

Abstract

Batch Normalization (BN) is a ubiquitous technique in deep learning, yet its theoretical convergence properties have remained largely unproven due to the non-gradient updates of its normalization parameters. This paper introduces Diminishing Batch Normalization (DBN), a generalization of BN that employs a diminishing moving average for updating mean and variance statistics. We provide the first theoretical convergence guarantee for a BN-like algorithm in non-convex settings, proving that DBN reaches a stationary point. Empirical results on MNIST, Network Intrusion, and CIFAR-10 demonstrate that DBN not only provides theoretical stability but also outperforms standard BN in practice.

Introduction: The "Black Box" of Batch Normalization

Batch Normalization has become a standard component in modern neural networks, celebrated for its ability to accelerate training and reduce internal covariate shift. However, its theoretical foundation is surprisingly sparse. The primary challenge in proving BN's convergence lies in its hybrid update logic: while weights are updated via gradient descent, the normalization parameters (mean and variance) are typically recalculated based on mini-batch statistics. This departure from pure gradient-based optimization invalidates classical convergence proofs.

The Diminishing Batch Normalization (DBN) Algorithm

To bridge this theoretical gap, we propose the DBN algorithm. DBN modifies the standard BN update by incorporating a weighted moving average of past statistics.

Algorithm 1: The DBN update procedure, showing the gradient update for weights and the weighted average update for normalization parameters.

The core innovation is the parameter $\alpha^{(m)}$, which controls the influence of the current batch's statistics. When $\alpha^{(m)} = 1$, the algorithm reverts to standard BN. By setting $\alpha$ to a diminishing sequence (e.g., $1/m$), the normalization layers become increasingly stable as training progresses, reflecting the distribution of the entire dataset rather than just the current mini-batch.

Theoretical Framework and Convergence

We analyze a two-layer neural network model with an arbitrary activation function and a non-convex objective. The architecture includes a hidden layer followed by a BN layer before the output.

Architecture of the two-layer model used for convergence analysis, highlighting the BN layer following the first hidden layer.

Our analysis proves that if the step sizes ($\eta$) and the diminishing weights ($\alpha$) satisfy specific summability conditions (square-summable but not summable), the algorithm is guaranteed to converge to a stationary point. This represents the first formal convergence proof for an algorithm incorporating BN layers in a non-convex setting.

Experimental Evaluation

We evaluated DBN against standard BN across three benchmarks:

  1. MNIST: Using a four-layer fully connected network (FNN), DBN ($\alpha=0.5$) achieved a test error of 1.84%, compared to 2.70% for standard BN.
  2. Network Intrusion (NI): On the KDD 1999 dataset, DBN ($\alpha=0.25$) reached an error of 7.24%, outperforming BN's 7.69%.
  3. CIFAR-10: In a complex CNN, DBN ($\alpha=0.001$) achieved a test error of 16.28%, a significant improvement over BN's 17.31%.

A key finding was the robustness of $\alpha = 0.25$, which consistently ranked among the top performers across all datasets. Conversely, setting $\alpha=0$ (no updates) led to catastrophic failure, emphasizing that while diminishing updates are beneficial, continuous adaptation to the data distribution is essential.

Conclusion and Future Work

DBN provides a mathematically grounded alternative to standard Batch Normalization that offers both theoretical guarantees and improved empirical performance. By stabilizing the normalization statistics through historical averages, DBN smooths the optimization landscape.

Future work will focus on extending the theoretical proof to deeper networks and stochastic gradient settings. While the current analysis assumes a deterministic "full gradient" approach to maintain mathematical tractability, the empirical success of DBN with stochastic gradients suggests that a broader proof is within reach.

Summary updated

Abstract

In this paper, we propose a generalization of the Batch Normalization (BN) algorithm, diminishing batch normalization (DBN), where we update the BN parameters in a diminishing moving average way. BN is very effective in accelerating the convergence of a neural network training phase that it has become a common practice. Our proposed DBN algorithm remains the overall structure of the original BN algorithm while introduces a weighted averaging update to some trainable parameters. We provide an analysis of the convergence of the DBN algorithm that converges to a stationary point with respect to trainable parameters. Our analysis can be easily generalized for original BN algorithm by setting some parameters to constant. To the best knowledge of authors, this analysis is the first of its kind for convergence with Batch Normalization introduced. We analyze a two-layer model with arbitrary activation function. The primary challenge of the analysis is the fact that some parameters are updated by gradient while others are not. The convergence analysis applies to any activation function that satisfies our common assumptions. In the numerical experiments, we test the proposed algorithm on complex modern CNN models with stochastic gradients and ReLU activation. We observe that DBN outperforms the original BN algorithm on MNIST, NI and CIFAR-10 datasets with reasonable complex FNN and CNN models.
BibTeX
@misc{ma2017convergence,
  author = {Yintai Ma and Diego Klabjan},
  title = {{Convergence Analysis of Batch Normalization for Deep Neural Nets}},
  year = {2017},
  eprint = {1705.08011},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1705.08011},
}
Figure