Abstract
Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training. Training with large batches can reduce these overheads; however, large batches can affect the convergence properties and generalization performance of SGD. In this work, we take a first step towards analyzing how the structure (width and depth) of a neural network affects the performance of large-batch training. We present new theoretical results which suggest that—for a fixed number of parameters—wider networks are more amenable to fast large-batch training compared to deeper ones. We provide extensive experiments on residual and fully-connected neural networks which suggest that wider networks can be trained using larger batches without incurring a convergence slow-down, unlike their deeper variants.
Introduction: The Large-Batch Bottleneck
In distributed deep learning, communication overhead is a primary bottleneck. Synchronizing gradients across multiple GPUs or nodes frequently slows down training. Increasing the batch size is a common strategy to reduce this frequency, but it often leads to a "critical batch size" threshold. Beyond this point, increasing the batch size further does not yield linear speedups; instead, the model requires more epochs to reach the same accuracy, wasting computational resources. This research investigates how architectural choices—specifically width and depth—influence this threshold.
Gradient Diversity: The Underlying Mechanism
The core concept introduced is Gradient Diversity ($\Delta_S(w)$), a metric measuring the dissimilarity between gradients of individual data points. High diversity implies that a large batch provides a wealth of non-redundant information, facilitating parallelization. Conversely, low diversity suggests redundancy, where a large batch offers little benefit over a smaller one.
As shown in the figure above, gradient diversity decreases as networks become deeper and narrower (moving from L:1 to L:10). Crucially, there is a strong correlation between gradient diversity and the largest batch size that can be used before convergence slows down. Wider layers effectively provide more "dimensions" for gradients to differ, acting as a diversity multiplier.
Empirical Validation
To test this theory, the researchers conducted experiments across various datasets (MNIST, CIFAR-10, EMNIST, Gisette) using fully-connected and residual networks. To ensure a fair comparison, they fixed the total number of parameters and traded depth for width.
The results consistently show that shallow, wide networks maintain stable convergence rates even at massive batch sizes. For example, a Depth 1 network can handle a batch size of 4,096 with minimal increase in training epochs. In contrast, a Depth 10 network of the same parameter count sees the required epochs skyrocket from approximately 3 to over 50 when scaled to the same batch size.
Key Insights
- Width as an Optimization Strategy: Traditionally, architectures are designed for accuracy. This work suggests designing for "parallelizability" by favoring wider structures to bypass communication bottlenecks in large-scale systems.
- The $B^*$ Threshold: Wider networks possess a significantly higher critical batch size ($B^*$), allowing for more efficient scaling in distributed environments.
- Depth as a Bottleneck: While depth is often associated with model capacity, it acts as a bottleneck for large-batch training by making gradients more redundant across the batch.
Conclusion and Future Work
This study establishes a fundamental link between network architecture and training efficiency. While the empirical evidence for nonlinear networks is robust, future work aims to formalize theoretical proofs for multi-layer nonlinear models. Additionally, exploring how these principles apply to convolutional filters and attention mechanisms in Transformers remains an open area for research.
Summary updated