Abstract
In large-scale distributed deep learning, the Parameter Server (PS) architecture often encounters significant network bottlenecks. While existing optimizations focus on the backward pass, the forward pass frequently remains sequential, forcing workers to wait for parameter pulls before starting computation. This paper introduces iBatch, a novel communication strategy that intelligently groups layers into batches to overlap parameter fetching with forward computation. By balancing the trade-off between communication startup overhead and computation time, iBatch improves training scalability by up to 73% compared to standard PS implementations.
The Parameter Server Bottleneck
Distributed training typically employs a Parameter Server (PS) architecture. In this model, "worker" nodes pull model parameters from central "servers," perform local computations, and then push updated gradients back to the servers. As clusters scale to dozens or hundreds of nodes, the network bandwidth becomes the primary performance ceiling.
Previous research has successfully overlapped the backward pass (gradient pushing) with computation. However, the forward pass has remained largely sequential. A naive approach to solve this—pulling parameters layer-by-layer—often fails because every network request incurs a "startup time" ($T_s$) overhead. In deep models with hundreds of layers, these cumulative delays can make distributed training slower than single-machine execution.
The iBatch Methodology
iBatch treats the grouping of layers for communication as an optimization problem. The system operates in four primary stages:
- Profiling: iBatch measures the computation time ($C_i$) and parameter transfer time ($T_t^i$) for every layer within the specific hardware environment, accounting for network latency.
- Optimization: The goal is to minimize total execution time. Pulling too many layers at once prevents early computation, while pulling too few increases the impact of startup overhead ($T_s$).
- Greedy Algorithms: The authors developed greedy algorithms to determine optimal "cut points" for batches. These algorithms maximize the overlap between the network transfer of batch $k+1$ and the computation of batch $k$.
- Implementation: Integrated into the BigDL framework, iBatch uses multi-threading to allow simultaneous parameter fetching and forward pass execution.
Key Insights
- Startup Time Penalty: Network latency is not just a function of data size; the number of requests matters. iBatch reduces the number of requests by batching small layers together.
- Heterogeneous Layer Demands: Neural networks consist of communication-heavy layers (like fully-connected layers) and computation-heavy layers (like convolutional layers). iBatch adapts its batching strategy to these varying demands rather than using a fixed batch size.
- Complementary Scalability: Because iBatch focuses on the forward pass, it can be combined with backward pass optimizations like gradient sparsification to achieve near-linear scalability.
Experimental Results
The researchers evaluated iBatch on a cluster of up to 72 nodes using the ImageNet22K and ILSVRC12 datasets across several architectures, including GoogLeNet and VGG19.
- Scalability Boost: On a 72-node cluster training VGG19-22K (229 million parameters), iBatch improved scalability by 73% over the default PS and 41% over the layer-by-layer strategy.
- Throughput Efficiency: For models like Inception-V3, iBatch maintained high throughput as the cluster size grew, whereas the default PS performance began to plateau or degrade due to network congestion.
- Speedup: The system achieved a 45x speedup on 72 nodes for the VGG19-22K network.
Conclusion and Future Work
iBatch demonstrates that intelligent communication batching is essential for scaling deep learning on commodity clusters. By addressing the forward pass bottleneck, it provides a significant performance leap for parameter-heavy models. Future work includes extending iBatch to frameworks like TensorFlow and developing dynamic profiling methods to adapt to fluctuating network conditions in shared cloud environments.
Summary updated