Fast Distributed Training of Deep Neural Networks: Dynamic Communication Thresholding for Model and Data Parallelism

Vipul Gupta, Dhruv Choudhary, Ping Tak Peter Tang, Xiaohan Wei, Xing Wang, Yuzhen Huang, Arun Kejariwal, Kannan Ramchandran, Michael W. Mahoney

Abstract

In this paper, we consider hybrid parallelism—a paradigm that employs both Data Parallelism (DP) and Model Parallelism (MP)—to scale distributed training of large recommendation models. We propose a compression framework called Dynamic Communication Thresholding (DCT) for communication-efficient hybrid training. DCT filters the entities to be communicated across the network through a simple hard-thresholding function, allowing only the most relevant information to pass through. For communication-efficient DP, DCT compresses the parameter gradients sent to the parameter server during model synchronization. The threshold is updated only once every few thousand iterations to reduce the computational overhead of compression. For communication-efficient MP, DCT incorporates a novel technique to compress the activations and gradients sent across the network during forward and backward propagation. We evaluate DCT on natural language processing and recommender models, including production systems at Facebook. DCT reduces communication by at least $100\times$ and $20\times$ during DP and MP, respectively, improving end-to-end training time for a state-of-the-art industrial recommender model by 37% without loss in performance.

Introduction: The Communication Bottleneck

Modern deep learning models, particularly recommendation systems (RecSys) and large language models, have grown to billions of parameters, exceeding the memory capacity of single machines. To train these models, engineers utilize Hybrid Parallelism (HP), which combines:

  1. Data Parallelism (DP): Multiple workers train on different batches of data and must synchronize their gradients.
  2. Model Parallelism (MP): The model is partitioned across different workers, requiring them to pass activations forward and gradients backward during every step.

As models scale, the network becomes a massive bottleneck. Existing compression techniques like quantization or sparsification often introduce high computational overhead (e.g., sorting millions of values to find the top 1%) that can negate the benefits of reduced network traffic.

Methods: Dynamic Communication Thresholding (DCT)

Hybrid parallelism architecture showing the interaction between the Parameter Server and multiple Trainers.

The proposed DCT framework targets both DP and MP communication channels to alleviate network congestion:

  • DCT-DP (Gradients): Instead of sending every gradient update to the parameter server, DCT applies a hard-thresholding function to send only the most significant values. To avoid the high cost of frequent sorting, DCT employs lazy thresholding, updating the threshold value only once every few thousand iterations.
  • DCT-MP (Activations): In model parallelism, DCT identifies and communicates only the most relevant neurons for each training sample. By transmitting only the top ~5% of activations, it significantly reduces the data transferred between model shards.
  • Error Compensation: To maintain accuracy, "ignored" gradient information is stored locally and accumulated into the next iteration, ensuring that small but consistent updates are eventually communicated.

Key Insights and Results

Threshold Stability

A core finding of this research is that the distribution of gradients remains remarkably stable over short periods of training. This stability justifies the "lazy" update strategy, allowing for high compression with minimal computational overhead.

Threshold stability over training iterations for various top-K percentages, showing consistent values over time.

Performance Gains

DCT was evaluated across several benchmarks, including the Criteo Ad Kaggle dataset (DLRM), NLP tasks (IWSLT’14 German-to-English), and Facebook’s internal production systems. Key results include:

  • Communication Reduction: Achieved a 100× reduction in DP and a 20× reduction in MP.
  • Training Speed: End-to-end training time for industrial models was reduced by 37%.
  • Regularization Effect: Sparsifying activations in MP did not degrade accuracy; in some cases, it acted as a "winner-take-all" regularizer, slightly improving performance.

Conclusion

Dynamic Communication Thresholding effectively shifts the distributed training bottleneck from the network to the processors. By leveraging the temporal stability of gradient distributions and employing error compensation, DCT enables massive scaling of recommendation and NLP models without sacrificing accuracy. Future work will explore layer-sensitive sparsification to further optimize communication efficiency.

Summary updated

Abstract

In this paper, we consider hybrid parallelism -- a paradigm that employs both Data Parallelism (DP) and Model Parallelism (MP) -- to scale distributed training of large recommendation models. We propose a compression framework called Dynamic Communication Thresholding (DCT) for communication-efficient hybrid training. DCT filters the entities to be communicated across the network through a simple hard-thresholding function, allowing only the most relevant information to pass through. For communication efficient DP, DCT compresses the parameter gradients sent to the parameter server during model synchronization. The threshold is updated only once every few thousand iterations to reduce the computational overhead of compression. For communication efficient MP, DCT incorporates a novel technique to compress the activations and gradients sent across the network during the forward and backward propagation, respectively. This is done by identifying and updating only the most relevant neurons of the neural network for each training sample in the data. We evaluate DCT on publicly available natural language processing and recommender models and datasets, as well as recommendation systems used in production at Facebook. DCT reduces communication by at least $100\times$ and $20\times$ during DP and MP, respectively. The algorithm has been deployed in production, and it improves end-to-end training time for a state-of-the-art industrial recommender model by 37\%, without any loss in performance.
BibTeX
@inproceedings{gupta2021fast,
  author = {Vipul Gupta and Dhruv Choudhary and Ping Tak Peter Tang and Xiaohan Wei and Xing Wang and Yuzhen Huang and Arun Kejariwal and Kannan Ramchandran and Michael W. Mahoney},
  title = {{Fast Distributed Training of Deep Neural Networks: Dynamic Communication Thresholding for Model and Data Parallelism}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2021},
  doi = {10.1145/3447548.3467080},
}
Figure