TL;DR
This paper introduces Dynamic Communication Thresholding (DCT), a novel framework for efficiently training large-scale recommender systems using hybrid parallelism (combining data and model parallelism). DCT significantly reduces communication overhead by selectively transmitting only the most important information, leading to faster training times without sacrificing accuracy.
What Problem Does This Paper Address?
Training massive deep learning models, particularly for recommendation systems, has become a significant challenge due to the sheer scale of the models and the data. Traditional approaches like Data Parallelism (DP) and Model Parallelism (MP) have limitations. DP struggles with increasing model sizes and the need for large batch sizes, which can hurt generalization. MP, while accommodating larger models, doesn't fully leverage large batch sizes. Hybrid Parallelism (HP) combines DP and MP to overcome these, but it introduces a new bottleneck: communication bandwidth. This paper addresses this communication bottleneck in HP training by proposing a method to compress the data exchanged between different parts of the distributed system.
Methods
The core contribution is the Dynamic Communication Thresholding (DCT) framework, which comprises two main components:
- DCT-DP (Data Parallelism): For DP, DCT compresses parameter gradients sent to the parameter server. It uses a hard-thresholding function to filter out less important gradient values. Crucially, the threshold for this filtering is updated infrequently (every few thousand iterations) to minimize computational overhead. This approach is inspired by existing sparsification techniques but focuses on practical implementation for real-world training times.
- DCT-MP (Model Parallelism): For MP, DCT introduces a novel technique to compress activations (during forward passes) and gradients (during backward passes) exchanged between model partitions. It identifies and transmits only the most relevant neurons for each training sample, effectively sparsifying the data. This is achieved by dynamically determining a threshold based on neuron activation values.
Both DCT-DP and DCT-MP aim to reduce communication by transmitting only a fraction of the data, with mechanisms to compensate for potential errors introduced by compression.
Datasets
The paper evaluates DCT on:
- Publicly available natural language processing (NLP) and recommender models and datasets.
- Recommendation systems used in production at Facebook.
- Specifically mentioned datasets include the Criteo Ad Kaggle Dataset for the Deep Learning Recommendation Model (DLRM) and the IWSLT'14 German to English dataset for a Transformer translation model.
Key Insights and Ideas
- Communication is the Bottleneck: The paper strongly emphasizes that as models scale, communication bandwidth, not computation, becomes the primary limiting factor in distributed training.
- Selective Information Transfer: The core idea is that not all information being communicated is equally important. By intelligently filtering or sparsifying this information, significant communication savings can be achieved.
- Infrequent Threshold Updates for DP: For Data Parallelism, updating the compression threshold only periodically (e.g., every few thousand iterations) is key to avoiding the overhead of frequent threshold calculations, which can negate the benefits of compression.
- Neuron Activation as a Sparsity Indicator for MP: In Model Parallelism, the activation values of neurons during a forward pass can serve as a proxy for their relevance. Sparsifying based on these activations allows for efficient compression without significant loss of information.
- DCT-MP Induces Regularization: The sparsity induced by DCT-MP in neuron activations can act as a form of implicit regularization, potentially leading to improved generalization performance.
- Error Compensation is Crucial: While compression introduces errors, the paper highlights the importance of error feedback mechanisms (for DP) and the inherent properties of the MP approach to ensure that model accuracy is maintained or even improved.
Key Results
- Significant Communication Reduction: DCT reduces communication by up to 100x for DP and 20x for MP.
- Improved End-to-End Training Time: For a state-of-the-art industrial recommender model, DCT improved end-to-end training time by 37%.
- No Loss in Performance: The proposed algorithms achieve these speedups without any loss in model performance (accuracy or loss).
- Shifted Bottleneck: DCT successfully shifts the training bottleneck from communication to computation, evidenced by a significant decrease in network utilization (from 94.2% to 49.3%) and an increase in CPU utilization (from 48.7% to 91.1%) in production systems.
- DCT-MP Sparsity: DCT-MP can achieve high sparsity levels (e.g., 95%) in activations.
- DCT-DP Robustness: DCT-DP can handle high sparsity levels (up to 98%) without performance degradation.
Limitations and Future Work
- Negative Results for Other Schemes: The paper presents negative results for Gaussian sketching for activation compression and basic top-K thresholding for gradient compression in MP, suggesting these methods are not as effective in this context. This highlights the importance of the specific design choices in DCT.
- DCT-MP Layer Sensitivity: The authors acknowledge that for DCT-MP, some layers might be more sensitive to sparsification than others. They suggest that identifying these sensitive layers and avoiding compression or adjusting split locations could further improve performance. This selectivity is noted as being beyond the scope of the current paper.
- Hyperparameter Tuning: The paper mentions that split locations for MP were not hyperparameter-tuned, implying that further optimization might be possible.
- Error Feedback in Asynchronous DP: The paper notes that error feedback works well in asynchronous DP, even though theoretical guarantees typically apply to synchronous SGD. They also mention a practical detail of draining the error feedback buffer to manage stale updates.
Summary updated