Theoretical Analysis of Auto Rate-Tuning by Batch Normalization

Sanjeev Arora, Zhiyuan Li, Kaifeng Lyu

Abstract

Batch Normalization (BN) has become a cornerstone of deep learning across diverse architectures, appearing to help optimization as well as generalization. While the idea makes intuitive sense, theoretical analysis of its effectiveness has been lacking. Here theoretical support is provided for one of its conjectured properties, namely, the ability to allow gradient descent to succeed with less tuning of learning rates. It is shown that even if we fix the learning rate of scale-invariant parameters (e.g., weights of each layer with BN) to a constant (say, $0.3$), gradient descent still approaches a stationary point (i.e., a solution where gradient is zero) in the rate of $T^{-1/2}$ in $T$ iterations, asymptotically matching the best bound for gradient descent with well-tuned learning rates. A similar result with convergence rate $T^{-1/4}$ is also shown for stochastic gradient descent.

Introduction

Since its introduction in 2015, Batch Normalization (BN) has become a "must-have" for deep learning, yet our theoretical understanding of why it works has remained murky. The original inventors suggested it reduces "Internal Covariate Shift," but recent studies have debunked that intuition.

One of the most celebrated practical benefits of BN is that it makes training less sensitive to the choice of learning rate. Practitioners often find they can use much higher learning rates with BN without the model exploding. This paper seeks to provide a formal mathematical foundation for this "auto rate-tuning" behavior. The authors ask: Can we rigorously prove that BN-equipped networks converge even when we do not tune the learning rate for the weights?

Theoretical Framework: Scale Invariance

The authors approach the problem by partitioning neural network parameters into two distinct groups:

  1. Scale-invariant parameters ($W$): These are the weights of layers followed by BN. Because BN standardizes the output, multiplying these weights by a constant $c$ does not change the network's output.
  2. Scale-variant parameters ($g$): These are parameters like the learnable gain ($\gamma$) and bias ($\beta$) in the BN layer itself, where scale actually matters.

The core of their method is a "Key Observation": for any scale-invariant weight, the weight vector and its gradient are always perpendicular. This geometric property leads to a phenomenon where the norm (length) of the weights grows monotonically during training. As the weights grow larger, the "effective" learning rate—the actual step size taken in the direction of the gradient—automatically decays.

The Mechanism of Auto-Tuning

This research highlights several elegant insights into how BN functions as a mathematical engine:

  • The Perpendicularity Principle: The gradient of a scale-invariant function is always orthogonal to the parameter vector. This is the engine that drives weight growth, which in turn drives the automatic decay of the learning rate.
  • Weight Growth as a Feature: In many optimization contexts, exploding weights are a sign of failure. Here, the authors show that weight growth is exactly how BN "tunes" itself. It acts as a built-in annealing schedule.
  • Equivalence of Initialization and Learning Rate: A surprising result is the mathematical equivalence between initialization and learning rate. The authors show that multiplying the initial scale of weights by a constant $C$ is mathematically identical to dividing the learning rate by $C^2$. This suggests that researchers should report their initialization scales as carefully as their learning rates.

Convergence Results

The authors use these geometric properties to prove convergence bounds for both full-batch Gradient Descent (GD) and Stochastic Gradient Descent (SGD) under the assumption that the loss function is smooth:

  • GD Convergence: In full-batch gradient descent, as long as the learning rate for the scale-variant parameters ($g$) is set reasonably, the network will converge to a stationary point at a rate of $O(T^{-1/2})$ regardless of how the learning rate for the weights ($W$) is set.
  • SGD Convergence: For stochastic settings, they prove a convergence rate of $O(T^{-1/4} \text{polylog}(T))$, which asymptotically matches the best-known bounds for well-tuned SGD.

Empirical Validation

While primarily a theoretical paper, the findings were validated using a modified VGGNet architecture. The experiments showed that while standard SGD with BN converges across a massive range of learning rates (from $10^{-1}$ to $10^2$), a "Projected" version (which manually keeps weight norms constant to remove the auto-tuning effect) fails to converge as soon as the learning rate becomes slightly too large.

Limitations and Future Work

Despite the rigorous proofs, some constraints remain. The current proof requires the loss function to be smooth (twice continuously differentiable), which technically excludes the popular ReLU activation function. Additionally, the theory still requires the learning rate for the scale-variant parameters ($\gamma$ and $\beta$) to be somewhat tuned. Finally, the paper focuses on reaching a "stationary point" (minimizing training loss); future work is needed to explore how this auto-tuning interacts with generalization and test accuracy.

Summary updated

Abstract

Batch Normalization (BN) has become a cornerstone of deep learning across diverse architectures, appearing to help optimization as well as generalization. While the idea makes intuitive sense, theoretical analysis of its effectiveness has been lacking. Here theoretical support is provided for one of its conjectured properties, namely, the ability to allow gradient descent to succeed with less tuning of learning rates. It is shown that even if we fix the learning rate of scale-invariant parameters (e.g., weights of each layer with BN) to a constant (say, $0.3$), gradient descent still approaches a stationary point (i.e., a solution where gradient is zero) in the rate of $T^{-1/2}$ in $T$ iterations, asymptotically matching the best bound for gradient descent with well-tuned learning rates. A similar result with convergence rate $T^{-1/4}$ is also shown for stochastic gradient descent.
BibTeX
@inproceedings{arora2019theoretical,
  author = {Sanjeev Arora and Zhiyuan Li and Kaifeng Lyu},
  title = {{Theoretical Analysis of Auto Rate-Tuning by Batch Normalization}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=QWErEU8GBS-H},
}
Figure