The Asynchronous Training Algorithm Based on Sampling and Mean Fusion for Distributed RNN

Dejiao Niu, Tianquan Liu, Tao Cai, Shijie ZhouIEEE Access 2019 · DOI

Abstract

Training large-scale deep neural networks with distributed implementations is an effective way to improve efficiency. However, high network communication cost for synchronizing gradients and parameters is a major bottleneck in distributed training. In this work, we propose an asynchronous training algorithm based on sampling and mean fusion for distributed recurrent neural networks (RNNs). In this architecture, multiple distributed neuron nodes and an interaction node work together to implement training. Synchronization overhead is reduced by a unique asynchronous sampling strategy amongst the distributed neuron nodes. To compensate for accuracy loss caused by asynchronous parameter updates, a mean fusion algorithm is proposed, where the interaction node averages all local parameters. We mathematically prove the convergence of the proposed algorithm and demonstrate significant speed gains with less than 1% average accuracy loss on language modeling benchmarks.

Introduction

Training Recurrent Neural Networks (RNNs) is notoriously slow because their sequential nature makes them difficult to parallelize. Unlike Convolutional Neural Networks (CNNs), where different parts of an image can be processed simultaneously, RNNs rely on the previous time step to calculate the current one. When training RNNs across multiple computers, systems often hit a "synchronization bottleneck" where every worker node must wait for all others to finish calculations before updating the model's parameters. This paper introduces an asynchronous training algorithm that uses "sampling" to let distributed nodes work independently and "mean fusion" to periodically sync their knowledge, achieving a 25% speedup with less than 1% loss in accuracy.

Proposed Methodology

The authors propose a two-pronged strategy based on a "neuron-centered" distributed architecture, where the model itself is split by its hidden neurons across multiple nodes.

The neuron-centric distributed architecture for RNN training, showing the interaction node connected to multiple distributed neuron nodes.

1. Asynchronous Sampling

In a typical distributed RNN, an interaction node waits to collect hidden states from all neuron nodes to calculate the final output and error. The proposed method ditches this requirement. Instead, each neuron node calculates a "local" version of the output and error based only on its own hidden state. This treats the global output as a sampling problem, allowing nodes to keep working without waiting for a global consensus at every single step, effectively bypassing the communication overhead.

2. Mean Fusion

Because these local updates are inherently biased (they only see a fraction of the model), the model parameters would eventually drift apart. To fix this, the authors introduce a "Mean Fusion" step. The interaction node periodically collects the parameters from all distributed neurons and averages them. This "fuses" the collective intelligence of the cluster back into each node, stabilizing the training process and pulling nodes back toward the global optimum.

Flowchart of the proposed asynchronous training algorithm featuring sampling and parameter revision steps.

The researchers also provided a mathematical proof to show that this method is guaranteed to converge, drawing inspiration from Averaged Stochastic Gradient Descent (ASGD).

Experimental Evaluation

The algorithm was evaluated on two well-known language modeling benchmarks:

  • Penn Tree Bank (PTB): A standard dataset (approx. 1M tokens) used to test initial performance and convergence.
  • MSR Sentence Completion Challenge Data (SCCD): A much larger dataset (approx. 225M tokens) used to test the algorithm's scalability.

Key Results

  • Significant Speedup: On the PTB dataset with 200 distributed neurons, the asynchronous method was 24.28% faster than the synchronous baseline by the 40th epoch.
  • Negligible Accuracy Trade-off: Across both datasets, the loss in accuracy (measured by Perplexity or classification rate) was consistently less than 1%.
  • Scalability: On the larger MSR SCCD dataset, the algorithm maintained its edge, showing speed improvements of up to 22.7% in early epochs and staying consistently faster than synchronous training as the model grew.

Conclusion and Future Work

The "sampling and mean fusion" approach effectively addresses the synchronization bottleneck in distributed RNN training. While the results are promising, the authors noted that framework overhead in Spark can diminish gains as model size increases relative to dataset size. Future work will explore automated tuning for the fusion coefficient ($\gamma$) and the performance of this logic on high-speed GPU clusters with NVLink to reveal even higher performance ceilings.

Summary updated

Abstract

Training of large scale deep neural networks with distributed implementations is an effective way to improve the efficiency. However, high network communication cost for synchronizing gradients and parameters is a major bottleneck in distributed training. In this work, we propose an asynchronous training algorithm based on sampling and mean fusion for distributed recurrent neural network (RNN). In distributed RNN, multiple distributed neuron nodes and an interaction node work together to implement the training. The synchronization overhead is reduced by a unique asynchronous sampling strategy amongst the distributed neuron nodes. Then, in order to make up for the accuracy loss caused by the asynchronous parameter update, a mean fusion algorithm is proposed, where the interaction node averages all local parameters from the distributed neurons. We mathematically prove the convergence of the proposed algorithm. Experimental verification is performed on two language modeling benchmark datasets. The results demonstrate significant speed gains for distributed RNN, while the accuracy loss is less than 1% on average.
BibTeX
@inproceedings{niu2019asynchronous,
  author = {Dejiao Niu and Tianquan Liu and Tao Cai and Shijie Zhou},
  title = {{The Asynchronous Training Algorithm Based on Sampling and Mean Fusion for Distributed RNN}},
  booktitle = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2939851},
}
Figure