TL;DR
This paper investigates the impact of simultaneously compressing both activations and gradients during model-parallel training. They find that gradients are more sensitive to compression than activations, and that TopK compression with $K=10\%$ offers a good balance. Interestingly, models trained with compression perform better during inference with compression, and error feedback methods don't improve convergence but do enable compression-free inference with minimal quality loss.
What Problem Does This Paper Address?
Training massive neural networks, like GPT-4, demands immense computational resources, often necessitating distributed training. Model parallelism, where the model is split across multiple machines, is a common strategy. However, communication between these machines can become a significant bottleneck, slowing down training. This paper tackles this challenge by exploring how to reduce communication overhead through simultaneous compression of both the activations (data passed forward) and gradients (data passed backward) in a model-parallel setting. They aim to understand the trade-offs between compression levels and model convergence and performance.
Methods
The core of this work lies in empirical evaluation of various compression techniques within a simulated model-parallel training environment. They don't train on actual distributed hardware but integrate compression directly into the model's pipeline stages.
Key methods explored include:
- Quantization: Reducing the precision of floating-point numbers to a fixed number of bits (e.g., 2, 4, 6, 8 bits).
- TopK Compression: A sparsification technique that retains only the $K\%$ largest values (in absolute magnitude) and zeroes out the rest. They test various $K$ values (e.g., 50%, 30%, 20%, 10%, 5%, 2%).
- Error Feedback (EF) and EF21: Techniques designed to compensate for compression-induced errors by incorporating them into subsequent communication rounds. They also introduce an "EF-mixed" approach for TopK.
- AQ-SGD: An approach that uses a variant of error feedback for activations, with a per-example buffer.
Crucially, they investigate compressing both activations and gradients simultaneously, and also explore the impact of applying compression during inference.
Datasets
The paper uses standard datasets for their experiments:
- CIFAR-10: A widely used dataset for image classification, paired with the ResNet-18 convolutional neural network.
- Wikitext-2: A dataset for language modeling, used for fine-tuning the pre-trained GPT-2 small model.
Key Insights and Ideas
- Gradients are More Sensitive: The paper's most significant finding is that gradients are considerably more sensitive to compression than activations. This means gradients require less aggressive compression to maintain good convergence.
- TopK Compression Sweet Spot: For TopK compression, $K=10\%$ appears to be a sweet spot, offering substantial compression without severely harming model convergence.
- Compression for Inference is Key (for TopK): Models trained with TopK compression perform better during inference when compression is also applied. This suggests that the model adapts to the compressed distribution during training, and removing compression during inference can disrupt its learned behavior.
- Error Feedback's Dual Role: While error feedback techniques (EF, EF21) don't seem to improve model convergence in their model-parallel setup, they are surprisingly effective at enabling inference without compression with minimal quality degradation. This is a valuable finding for deployment scenarios.
- AQ-SGD with TopK Limitations: The AQ-SGD approach, when combined with TopK compression (instead of quantization as in the original work), did not yield improved convergence, especially with stronger TopK compression ($K < 30\%$).
Key Results
- Quantization: Gradients need at least 6 bits for acceptable test accuracy, while activations can be quantized to 2 or 4 bits.
- TopK Compression:
- For ResNet-18 on CIFAR-10, Top10% compression (both activations and gradients) achieved test accuracy comparable to uncompressed models when compression was also applied during inference.
- For GPT-2 fine-tuning on Wikitext, TopK compression was more detrimental, with Top20% already significantly increasing validation loss. Top10% separate compression (activations and gradients independently) led to severe performance degradation.
- Error Feedback:
- EF and EF21 with TopK compression did not improve convergence over plain TopK.
- However, EF techniques made test accuracy results for compressed inference very close to uncompressed inference (only 1-2 percentage points difference).
- AQ-SGD: AQ-SGD with TopK compression did not improve convergence compared to plain TopK compression.
Limitations and Future Work
The authors acknowledge several limitations:
- Lack of Large-Scale Experiments: The experiments were conducted on ResNet-18 and GPT-2 small, not state-of-the-art large models, which might behave differently.
- Limited Hyperparameter Tuning: They tested only one model parallelism pipeline and a limited set of hyperparameters for each experiment.
Future research directions suggested include:
- Experiments with larger language models.
- Reducing the memory footprint of the AQ-SGD approach.
- Exploring more biased compression techniques beyond TopK.
- Investigating the impact of architectural differences (CNNs vs. Transformers) on compression effectiveness.
Summary updated