Diminishing Batch Normalization

Yintai Ma, Diego Klabjan

Abstract

In this paper, we propose a generalization of the Batch Normalization (BN) algorithm, diminishing batch normalization (DBN), where we update the BN parameters in a diminishing moving average way. BN is very effective in accelerating the convergence of a neural network training phase that it has become a common practice. Our proposed DBN algorithm remains the overall structure of the original BN algorithm while introduces a weighted averaging update to some trainable parameters. We provide an analysis of the convergence of the DBN algorithm that converges to a stationary point with respect to trainable parameters. Our analysis can be easily generalized for original BN algorithm by setting some parameters to constant. To the best knowledge of authors, this analysis is the first of its kind for convergence with Batch Normalization introduced. We analyze a two-layer model with arbitrary activation function. The primary challenge of the analysis is the fact that some parameters are updated by gradient while others are not. The convergence analysis applies to any activation function that satisfies our common assumptions. In the numerical experiments, we test the proposed algorithm on complex modern CNN models with stochastic gradients and ReLU activation. We observe that DBN outperforms the original BN algorithm on MNIST, NI and CIFAR-10 datasets with reasonable complex FNN and CNN models.

Introduction

Batch Normalization (BN) is a cornerstone of modern deep learning, essential for training deep networks by reducing "internal covariate shift." However, despite its massive empirical success, BN has long been a theoretical black box. Standard convergence analysis for gradient descent assumes all parameters are updated via gradients. In BN, the mean and variance parameters are updated by calculating batch statistics, not gradients, which invalidates classical proofs. Furthermore, in standard BN, the output for a specific training example depends on the other examples in the same mini-batch, introducing noise and instability during training.

The Diminishing Batch Normalization (DBN) Algorithm

To address these issues, we propose Diminishing Batch Normalization (DBN). While DBN maintains the core structure of BN, it fundamentally changes how the normalization parameters ($\mu$ and $\sigma$) are updated.

Neural network architecture with a Batch Normalization layer.

Instead of relying solely on the current mini-batch, DBN utilizes a weighted moving average that incorporates the history of all past mini-batches. The weight assigned to the current batch, $\alpha^{(m)}$, "diminishes" over time (e.g., following a schedule like $1/m$ or $1/m^2$). This approach ensures that the normalization becomes increasingly representative of the entire dataset rather than just the latest noisy mini-batch.

The formal DBN update algorithm.

Standard BN can be viewed as a special case of DBN where the weight $\alpha$ is always set to 1. By introducing the diminishing weight, we bridge the gap between gradient-based weight updates and recalculation-based parameter updates within a single mathematical framework.

Convergence Analysis

A key contribution of this work is the first theoretical convergence guarantee for a BN-style algorithm. We analyze a two-layer non-convex model with arbitrary activation functions. We prove that under specific conditions for the step sizes and diminishing weights, the algorithm is guaranteed to converge to a stationary point. This analysis is significant because it accounts for the hybrid nature of the updates—where some parameters follow gradients and others follow batch statistics.

Experimental Results

We evaluated DBN against standard BN using three benchmark datasets: MNIST, Network Intrusion (NI) KDD 1999, and CIFAR-10.

  • MNIST: Using a four-layer fully connected network (FNN), DBN ($\alpha=0.5$) achieved a 1.84% error rate, compared to 2.70% for standard BN.
  • Network Intrusion: On a four-layer FNN, DBN ($\alpha=0.25$) achieved 7.24% error, outperforming standard BN's 7.69%.
  • CIFAR-10: Using a complex CNN, DBN ($\alpha=0.001$) achieved 16.28% error, compared to 17.31% for standard BN.

Qualitatively, DBN exhibited significantly more stability. Because it does not "overreact" to the statistics of a single mini-batch, the variance in performance across training epochs is much lower, resulting in smoother validation loss curves. The authors noted that $\alpha = 0.25$ was a particularly robust choice, performing in the top three across all tested datasets.

Conclusion and Future Work

DBN provides a theoretically grounded and empirically superior alternative to standard Batch Normalization. While our current convergence analysis is limited to a two-layer model and assumes full gradient updates, the results on deep CNNs suggest broad applicability. Future work will focus on extending the theoretical proofs to deeper networks and the stochastic gradient setting, as well as exploring DBN's application in other complex architectural components like "wrapped layers."

Summary updated

Abstract

In this paper, we propose a generalization of the Batch Normalization (BN) algorithm, diminishing batch normalization (DBN), where we update the BN parameters in a diminishing moving average way. BN is very effective in accelerating the convergence of a neural network training phase that it has become a common practice. Our proposed DBN algorithm remains the overall structure of the original BN algorithm while introduces a weighted averaging update to some trainable parameters. We provide an analysis of the convergence of the DBN algorithm that converges to a stationary point with respect to trainable parameters. Our analysis can be easily generalized for original BN algorithm by setting some parameters to constant. To the best knowledge of authors, this analysis is the first of its kind for convergence with Batch Normalization introduced. We analyze a two-layer model with arbitrary activation function. The primary challenge of the analysis is the fact that some parameters are updated by gradient while others are not. The convergence analysis applies to any activation function that satisfies our common assumptions. In the numerical experiments, we test the proposed algorithm on complex modern CNN models with stochastic gradients and ReLU activation. We observe that DBN outperforms the original BN algorithm on MNIST, NI and CIFAR-10 datasets with reasonable complex FNN and CNN models.
BibTeX
@misc{ma2017diminishing,
  author = {Yintai Ma and Diego Klabjan},
  title = {{Diminishing Batch Normalization}},
  year = {2017},
  eprint = {1705.08011},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1705.08011},
}
Figure