PSO-PS: Parameter Synchronization with Particle Swarm Optimization for Distributed Training of Deep Neural Networks

Qing Ye, Yuxuan Han, Yanan Sun, Jiancheng Lv

Introduction

Distributed deep learning relies on data parallelism to handle massive datasets by distributing them across multiple worker nodes. The primary challenge in this setup is parameter synchronization. The standard approach, Synchronous Stochastic Gradient Descent (SSGD), requires all workers to average their gradients or weights at specific intervals. However, this synchronization creates a communication bottleneck, particularly due to the "straggler" problem where the slowest node dictates the system's overall speed. While reducing synchronization frequency can mitigate communication overhead, it often degrades model convergence and accuracy.

The PSO-PS Framework

To address these challenges, this paper proposes PSO-PS (Particle Swarm Optimization - Parameter Synchronization). This method reimagines the parameter synchronization process as an evolutionary search problem. Instead of simple mathematical averaging, PSO-PS treats each worker node as a "particle" within a swarm.

Architecture of the PSO-PS parameter synchronization framework, illustrating the interaction between the Parameter Server and distributed worker nodes.

In this framework, the weights of the Deep Neural Network (DNN) represent the particle's position in a high-dimensional space, and the training loss on a local mini-batch serves as its fitness value. The Parameter Server manages the synchronization by updating weights based on three core PSO components:

  1. Inertia: The particle's previous velocity, maintaining momentum in weight updates.
  2. Local Best ($pBest$): The most successful weight configuration found by an individual worker.
  3. Global Best ($gBest$): The best weight configuration discovered by any worker in the entire cluster.

To stabilize training for complex neural networks, the authors implemented a linear inertia weight decay and a randomness adjustment parameter ($\lambda$). These refinements allow for broad exploration in early epochs and precise exploitation as training progresses, effectively mimicking a decaying learning rate.

Experimental Results

The researchers evaluated PSO-PS against the baseline SSGD using two benchmarks: MNIST (with LeNet) and CIFAR-10 (with ResNet).

Performance and Scalability

The results indicate that PSO-PS consistently outperforms SSGD, particularly as the number of worker nodes increases. On the MNIST dataset, PSO-PS achieved a peak accuracy of 99.0% with 4 workers, compared to 98.25% for SSGD.

Comparison of test error rates between SSGD and PSO-PS across different cluster scales (n=4, 8, and 16). PSO-PS demonstrates superior convergence and lower error rates as the scale increases.

As shown in the figure above, PSO-PS maintains its performance advantage even at a scale of 16 nodes, achieving approximately 97.93% accuracy while SSGD drops to 97.0%. This suggests that the "swarm intelligence" of the algorithm scales effectively with larger clusters.

Convergence and Efficiency

On the more complex CIFAR-10 task using ResNet, PSO-PS showed significantly faster convergence during the initial training phases. Crucially, the algorithm remains robust even when the synchronization "step" is increased. This allows for less frequent communication between nodes without the typical loss in accuracy associated with infrequent SSGD updates, thereby improving overall training efficiency.

Conclusion

PSO-PS offers a promising alternative to traditional gradient averaging in distributed training. By leveraging collective intelligence, it achieves better convergence stability and higher accuracy, especially in communication-constrained environments. Future work will focus on optimizing PSO-PS for even higher-dimensional parameter spaces and testing its performance on larger-scale distributed clusters.

Summary updated

Abstract

Parameter updating is an important stage in parallelism-based distributed deep learning. Synchronous methods are widely used in distributed training the Deep Neural Networks (DNNs). To reduce the communication and synchronization overhead of synchronous methods, decreasing the synchronization frequency (e.g., every n mini-batches) is a straightforward approach. However, it often suffers from poor convergence. In this paper, we propose a new algorithm of integrating Particle Swarm Optimization (PSO) into the distributed training process of DNNs to automatically compute new parameters. In the proposed algorithm, a computing work is encoded by a particle, the weights of DNNs and the training loss are modeled by the particle attributes. At each synchronization stage, the weights are updated by PSO from the sub weights gathered from all workers, instead of averaging the weights or the gradients. To verify the performance of the proposed algorithm, the experiments are performed on two commonly used image classification benchmarks: MNIST and CIFAR10, and compared with the peer competitors at multiple different synchronization configurations. The experimental results demonstrate the competitiveness of the proposed algorithm.
BibTeX
@inproceedings{ye2020pso,
  author = {Qing Ye and Yuxuan Han and Yanan Sun and Jiancheng Lv},
  title = {{PSO-PS: Parameter Synchronization with Particle Swarm Optimization for Distributed Training of Deep Neural Networks}},
  booktitle = {IJCNN},
  year = {2020},
  doi = {10.1109/ijcnn48605.2020.9207698},
}
Figure