Introduction
Distributed deep learning relies on data parallelism to handle massive datasets by distributing them across multiple worker nodes. The primary challenge in this setup is parameter synchronization. The standard approach, Synchronous Stochastic Gradient Descent (SSGD), requires all workers to average their gradients or weights at specific intervals. However, this synchronization creates a communication bottleneck, particularly due to the "straggler" problem where the slowest node dictates the system's overall speed. While reducing synchronization frequency can mitigate communication overhead, it often degrades model convergence and accuracy.
The PSO-PS Framework
To address these challenges, this paper proposes PSO-PS (Particle Swarm Optimization - Parameter Synchronization). This method reimagines the parameter synchronization process as an evolutionary search problem. Instead of simple mathematical averaging, PSO-PS treats each worker node as a "particle" within a swarm.
In this framework, the weights of the Deep Neural Network (DNN) represent the particle's position in a high-dimensional space, and the training loss on a local mini-batch serves as its fitness value. The Parameter Server manages the synchronization by updating weights based on three core PSO components:
- Inertia: The particle's previous velocity, maintaining momentum in weight updates.
- Local Best ($pBest$): The most successful weight configuration found by an individual worker.
- Global Best ($gBest$): The best weight configuration discovered by any worker in the entire cluster.
To stabilize training for complex neural networks, the authors implemented a linear inertia weight decay and a randomness adjustment parameter ($\lambda$). These refinements allow for broad exploration in early epochs and precise exploitation as training progresses, effectively mimicking a decaying learning rate.
Experimental Results
The researchers evaluated PSO-PS against the baseline SSGD using two benchmarks: MNIST (with LeNet) and CIFAR-10 (with ResNet).
Performance and Scalability
The results indicate that PSO-PS consistently outperforms SSGD, particularly as the number of worker nodes increases. On the MNIST dataset, PSO-PS achieved a peak accuracy of 99.0% with 4 workers, compared to 98.25% for SSGD.
As shown in the figure above, PSO-PS maintains its performance advantage even at a scale of 16 nodes, achieving approximately 97.93% accuracy while SSGD drops to 97.0%. This suggests that the "swarm intelligence" of the algorithm scales effectively with larger clusters.
Convergence and Efficiency
On the more complex CIFAR-10 task using ResNet, PSO-PS showed significantly faster convergence during the initial training phases. Crucially, the algorithm remains robust even when the synchronization "step" is increased. This allows for less frequent communication between nodes without the typical loss in accuracy associated with infrequent SSGD updates, thereby improving overall training efficiency.
Conclusion
PSO-PS offers a promising alternative to traditional gradient averaging in distributed training. By leveraging collective intelligence, it achieves better convergence stability and higher accuracy, especially in communication-constrained environments. Future work will focus on optimizing PSO-PS for even higher-dimensional parameter spaces and testing its performance on larger-scale distributed clusters.
Summary updated