Training Recommender Systems at Scale: Communication-Efficient Model and\n Data Parallelism

Vipul Gupta, Dhruv Choudhary, Ping Tang, Xiaohan Wei, Xing Wang, Yuzhen Huang, Arun Kejariwal, Kannan Ramchandran, Michael W. Mahoney

TL;DR

This paper introduces Dynamic Communication Thresholding (DCT), a novel framework for efficiently training large-scale recommender systems using hybrid parallelism (combining data and model parallelism). DCT significantly reduces communication overhead by selectively transmitting only the most important information, leading to faster training times without sacrificing accuracy.

What Problem Does This Paper Address?

Training massive deep learning models, particularly for recommendation systems, has become a significant challenge due to the sheer scale of the models and the data. Traditional approaches like Data Parallelism (DP) and Model Parallelism (MP) have limitations. DP struggles with increasing model sizes and the need for large batch sizes, which can hurt generalization. MP, while accommodating larger models, doesn't fully leverage large batch sizes. Hybrid Parallelism (HP) combines DP and MP to overcome these, but it introduces a new bottleneck: communication bandwidth. This paper addresses this communication bottleneck in HP training by proposing a method to compress the data exchanged between different parts of the distributed system.

Methods

The core contribution is the Dynamic Communication Thresholding (DCT) framework, which comprises two main components:

  • DCT-DP (Data Parallelism): For DP, DCT compresses parameter gradients sent to the parameter server. It uses a hard-thresholding function to filter out less important gradient values. Crucially, the threshold for this filtering is updated infrequently (every few thousand iterations) to minimize computational overhead. This approach is inspired by existing sparsification techniques but focuses on practical implementation for real-world training times.
  • DCT-MP (Model Parallelism): For MP, DCT introduces a novel technique to compress activations (during forward passes) and gradients (during backward passes) exchanged between model partitions. It identifies and transmits only the most relevant neurons for each training sample, effectively sparsifying the data. This is achieved by dynamically determining a threshold based on neuron activation values.

Both DCT-DP and DCT-MP aim to reduce communication by transmitting only a fraction of the data, with mechanisms to compensate for potential errors introduced by compression.

Datasets

The paper evaluates DCT on:

  • Publicly available natural language processing (NLP) and recommender models and datasets.
  • Recommendation systems used in production at Facebook.
  • Specifically mentioned datasets include the Criteo Ad Kaggle Dataset for the Deep Learning Recommendation Model (DLRM) and the IWSLT'14 German to English dataset for a Transformer translation model.

Key Insights and Ideas

  • Communication is the Bottleneck: The paper strongly emphasizes that as models scale, communication bandwidth, not computation, becomes the primary limiting factor in distributed training.
  • Selective Information Transfer: The core idea is that not all information being communicated is equally important. By intelligently filtering or sparsifying this information, significant communication savings can be achieved.
  • Infrequent Threshold Updates for DP: For Data Parallelism, updating the compression threshold only periodically (e.g., every few thousand iterations) is key to avoiding the overhead of frequent threshold calculations, which can negate the benefits of compression.
  • Neuron Activation as a Sparsity Indicator for MP: In Model Parallelism, the activation values of neurons during a forward pass can serve as a proxy for their relevance. Sparsifying based on these activations allows for efficient compression without significant loss of information.
  • DCT-MP Induces Regularization: The sparsity induced by DCT-MP in neuron activations can act as a form of implicit regularization, potentially leading to improved generalization performance.
  • Error Compensation is Crucial: While compression introduces errors, the paper highlights the importance of error feedback mechanisms (for DP) and the inherent properties of the MP approach to ensure that model accuracy is maintained or even improved.

Key Results

  • Significant Communication Reduction: DCT reduces communication by up to 100x for DP and 20x for MP.
  • Improved End-to-End Training Time: For a state-of-the-art industrial recommender model, DCT improved end-to-end training time by 37%.
  • No Loss in Performance: The proposed algorithms achieve these speedups without any loss in model performance (accuracy or loss).
  • Shifted Bottleneck: DCT successfully shifts the training bottleneck from communication to computation, evidenced by a significant decrease in network utilization (from 94.2% to 49.3%) and an increase in CPU utilization (from 48.7% to 91.1%) in production systems.
  • DCT-MP Sparsity: DCT-MP can achieve high sparsity levels (e.g., 95%) in activations.
  • DCT-DP Robustness: DCT-DP can handle high sparsity levels (up to 98%) without performance degradation.

Limitations and Future Work

  • Negative Results for Other Schemes: The paper presents negative results for Gaussian sketching for activation compression and basic top-K thresholding for gradient compression in MP, suggesting these methods are not as effective in this context. This highlights the importance of the specific design choices in DCT.
  • DCT-MP Layer Sensitivity: The authors acknowledge that for DCT-MP, some layers might be more sensitive to sparsification than others. They suggest that identifying these sensitive layers and avoiding compression or adjusting split locations could further improve performance. This selectivity is noted as being beyond the scope of the current paper.
  • Hyperparameter Tuning: The paper mentions that split locations for MP were not hyperparameter-tuned, implying that further optimization might be possible.
  • Error Feedback in Asynchronous DP: The paper notes that error feedback works well in asynchronous DP, even though theoretical guarantees typically apply to synchronous SGD. They also mention a practical detail of draining the error feedback buffer to manage stale updates.

Summary updated

Abstract

In this paper, we consider hybrid parallelism -- a paradigm that employs both\nData Parallelism (DP) and Model Parallelism (MP) -- to scale distributed\ntraining of large recommendation models. We propose a compression framework\ncalled Dynamic Communication Thresholding (DCT) for communication-efficient\nhybrid training. DCT filters the entities to be communicated across the network\nthrough a simple hard-thresholding function, allowing only the most relevant\ninformation to pass through. For communication efficient DP, DCT compresses the\nparameter gradients sent to the parameter server during model synchronization.\nThe threshold is updated only once every few thousand iterations to reduce the\ncomputational overhead of compression. For communication efficient MP, DCT\nincorporates a novel technique to compress the activations and gradients sent\nacross the network during the forward and backward propagation, respectively.\nThis is done by identifying and updating only the most relevant neurons of the\nneural network for each training sample in the data. We evaluate DCT on\npublicly available natural language processing and recommender models and\ndatasets, as well as recommendation systems used in production at Facebook. DCT\nreduces communication by at least $100\\times$ and $20\\times$ during DP and MP,\nrespectively. The algorithm has been deployed in production, and it improves\nend-to-end training time for a state-of-the-art industrial recommender model by\n37\\%, without any loss in performance.\n
BibTeX
@inproceedings{gupta2021training,
  author = {Vipul Gupta and Dhruv Choudhary and Ping Tang and Xiaohan Wei and Xing Wang and Yuzhen Huang and Arun Kejariwal and Kannan Ramchandran and Michael W. Mahoney},
  title = {{Training Recommender Systems at Scale: Communication-Efficient Model and\textbackslash\{\}n Data Parallelism}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2021},
  doi = {10.1145/3447548.3467080},
}
Figure