Extended Batch Normalization

Chunjie Luo, Jianfeng Zhan, Lei Wang, Wanling GaoarXiv 2020 · arXiv

TL;DR

This paper introduces Extended Batch Normalization (EBN), a simple modification to Batch Normalization (BN) that significantly improves its performance with small batch sizes. EBN achieves this by computing the standard deviation over a larger set of pixels, allowing it to maintain BN's benefits while mitigating its key weakness.

What Problem Does This Paper Address?

Modern deep neural networks rely heavily on Batch Normalization (BN) to accelerate training and improve performance. BN works by normalizing layer activations using the mean and standard deviation computed across a mini-batch of training examples. This approach has a critical drawback: its effectiveness degrades significantly when the batch size is small. This is because small batches lead to noisy and inaccurate estimations of the batch statistics, hindering training. This limitation is particularly problematic for training larger models that require small batches due to memory constraints, and for deploying models on resource-limited edge devices where small batch sizes are often necessary.

Methods

The paper proposes Extended Batch Normalization (EBN) as a solution. EBN builds upon the standard BN formulation but modifies how the statistics are computed. For NCHW format feature maps (Batch, Channel, Height, Width), EBN computes the mean along the (N, H, W) dimensions, just like standard BN, preserving its advantages. The key innovation lies in how the standard deviation is calculated: EBN computes it along the (N, C, H, W) dimensions. This effectively enlarges the set of pixels from which the standard deviation is estimated, making it more robust to variations caused by small batch sizes.

At inference time, EBN, like BN, uses pre-computed statistics (via moving average during training) which allows for efficient fusion into convolutional operations, speeding up deployment.

Datasets

The authors evaluated EBN on the following datasets:

  • MNIST
  • CIFAR-10/100
  • STL-10
  • ImageNet

Key Insights and Ideas

  • The "noise" in small batches is a double-edged sword: While BN's reliance on batch statistics makes it sensitive to noise from small batches, EBN leverages a larger statistical pool to smooth out this noise for standard deviation calculation.
  • Mean vs. Standard Deviation Sensitivity: EBN hypothesizes that the mean is less sensitive to batch size variations than the standard deviation. By keeping the mean calculation similar to BN and expanding the standard deviation calculation, they target the core issue.
  • Inference Efficiency: The ability to pre-compute statistics for inference means EBN doesn't introduce any additional computational overhead compared to standard BN during deployment.

Key Results

The experiments demonstrate that EBN consistently outperforms standard BN when using small batch sizes across various datasets and network architectures (fully-connected networks, ResNet18, VGG16).

  • MNIST: With a batch size of 4, EBN achieved the most stable and highest accuracy (98.07%), significantly outperforming BN (94.57%).
  • CIFAR-10: For ResNet18 with a batch size of 4, EBN achieved the best test accuracy (94.92%), while BN's accuracy fluctuated.
  • CIFAR-100: For VGG16 with a batch size of 4, EBN achieved the best test accuracy (72.66%), a notable improvement over BN (71.73%).
  • STL-10: With a batch size of 2, EBN achieved the highest accuracy (77.96%), outperforming BN (76.91%).
  • ImageNet: With a small batch size of 4 per GPU, EBN achieved 68.54% accuracy, close to GN (69.08%) and significantly better than BN (65.78%).

Importantly, EBN also performs comparably to standard BN when using large batch sizes, indicating it doesn't sacrifice performance in scenarios where BN already excels.

Limitations and Future Work

  • Comparison with Group Normalization: While EBN generally performs well, it's noted that Group Normalization (GN) can sometimes achieve better results, especially on datasets like CIFAR-100 with small batch sizes. The paper suggests that EBN's performance is "close" to GN in some small batch scenarios, implying room for further improvement.
  • Inconsistency between Training and Evaluation: The authors acknowledge that, similar to BN, EBN relies on moving averages for inference statistics. They plan to investigate the potential inconsistency between training and evaluation statistics, which has been shown to affect BN's performance with small batches.
  • Extensibility: The paper mentions that EBN could be combined with other techniques like Kalman normalization or switchable normalization, suggesting avenues for future research to further enhance its capabilities.

Summary updated

BibTeX
@misc{luo2020extended,
  author = {Chunjie Luo and Jianfeng Zhan and Lei Wang and Wanling Gao},
  title = {{Extended Batch Normalization}},
  year = {2020},
  eprint = {2003.05569},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.05569},
}
Figure