Abstract
To mitigate communication overheads in distributed model training, several studies propose the use of compressed stochastic gradients, usually achieved by sparsification or quantization. Such techniques achieve high compression ratios, but in many cases incur either significant computational overheads or some accuracy loss. In this work, we present Pufferfish, a communication and computation efficient distributed training framework that incorporates the gradient compression into the model training process via training low-rank, pre-factorized deep networks. Pufferfish not only reduces communication, but also completely bypasses any computation overheads related to compression, and achieves the same accuracy as state-of-the-art, off-the-shelf deep models. Pufferfish can be directly integrated into current deep learning frameworks with minimum implementation modification. Our extensive experiments over real distributed setups, across a variety of large-scale machine learning tasks, indicate that Pufferfish achieves up to 1.64x end-to-end speedup over the latest distributed training API in PyTorch without accuracy loss. Compared to the Lottery Ticket Hypothesis models, Pufferfish leads to equally accurate, small-parameter models while avoiding the burden of "winning the lottery". Pufferfish also leads to more accurate and smaller models than SOTA structured model pruning methods.
The Communication Bottleneck in Distributed Training
In distributed "data-parallel" training, multiple worker nodes compute gradients on different chunks of data and then synchronize those gradients. As models scale to billions of parameters, this synchronization becomes a massive bottleneck; the time spent moving data across the network often outweighs the time spent computing it.
Existing solutions usually involve gradient compression (like quantization or sparsification). However, these come with "hidden costs":
- Computational Overhead: Methods like ATOMO require performing an expensive Singular Value Decomposition (SVD) on every single batch.
- Memory Overhead: Techniques like "error feedback" require storing extra information to compensate for compression errors.
- Accuracy Loss: Many compression schemes struggle to maintain the original model's performance, especially on large-scale tasks.
The authors of Pufferfish ask: What if we didn't compress the gradients of a large model, but instead trained a smaller, factorized model that is mathematically designed to be communication-efficient from the start?
Methods: Architecture Factorization
Pufferfish shifts the focus from compressing gradients to factorizing the model architecture itself.
1. Low-Rank Factorization
The core idea is to replace large weight matrices ($W$) with two smaller "factors" ($U$ and $V$) such that $W \approx UV^T$.
- For Fully Connected (FC) layers: A large matrix is split into two thin ones.
- For Convolutional layers: A standard convolution is replaced by a "thinner" convolution followed by a $1 \times 1$ projection layer.
- For LSTMs and Transformers: The weight matrices within the gates and attention heads are similarly factorized.
2. The "Pufferfish" Training Recipe
Training low-rank models from scratch usually leads to poor accuracy. To fix this, the authors introduce two critical strategies:
- Vanilla Warm-up: The model starts as a standard, full-rank network. It is trained normally for a small fraction of time (e.g., the first 10% of epochs). This allows the model to find a good "basin" in the loss landscape.
- SVD Initialization: After the warm-up, the full-rank weights are decomposed via SVD to initialize the $U$ and $V$ factors of the low-rank model. Training then continues in this low-rank space.
- Hybrid Architecture: The authors found that the very first and very last layers of a network are highly sensitive. Pufferfish keeps these layers full-rank while factorizing the massive middle layers where most of the communication cost resides.
Key Results
Pufferfish was evaluated across Computer Vision (CIFAR-10, ImageNet), Language Modeling (WikiText-2), and Machine Translation (WMT 2016) tasks.
- End-to-End Speedup: Pufferfish achieved a 1.64× speedup over the standard PyTorch DistributedDataParallel (DDP) API for ResNet-50 on ImageNet without losing accuracy.
- Accuracy Parity: As shown in the training trajectory, Pufferfish matches the final top-1 accuracy of the full-rank baseline while maintaining a lower communication footprint throughout the majority of the training process.
- Efficiency vs. Baselines: Compared to PowerSGD (a popular low-rank gradient compressor), Pufferfish was faster because it completely skipped the per-batch encoding/decoding steps.
- Comparison to Pruning and LTH: Pufferfish reached the same compression levels as the Early Bird Ticket method but with 1.76% higher top-1 accuracy on ImageNet. When compared to the Lottery Ticket Hypothesis (LTH), Pufferfish was 5.67× faster to reach the same model size and accuracy because it avoids the expensive "train-prune-rewind" cycle.
Discussion and Future Work
Pufferfish demonstrates that communication-computation synergy is possible: factorized layers have fewer parameters to send and fewer floating-point operations (FLOPs) to compute. Interestingly, on Transformer tasks, the Pufferfish model actually outperformed the vanilla model in BLEU score, suggesting the low-rank constraint acts as a beneficial form of implicit regularization.
Future work includes using Neural Architecture Search (NAS) to find optimal ranks for each specific layer rather than using a global rank ratio, potentially leading to even more efficient models.
Summary updated