Abstract
Training large-scale deep learning models is frequently limited by the communication overhead of synchronizing gradients across distributed workers. We present Quantized Compressive Sampling (QCS), a novel technique that integrates random signal mixing with dithered quantization to achieve significant gradient compression (32x or more) while preserving model accuracy. By transforming compression noise into a manageable Gaussian-like distribution, QCS enables the use of larger learning rates and achieves faster convergence compared to existing state-of-the-art quantization and sparsification methods.
Introduction
In distributed deep learning, multiple "workers" (e.g., GPUs) compute local gradients and must synchronize updates to maintain a global model. As the scale of these systems increases, the "communication bottleneck"—the time required to transmit high-dimensional gradient vectors across the network—becomes the primary performance constraint.
Existing approaches to mitigate this bottleneck include quantization (reducing numerical precision) and sparsification (transmitting only a subset of gradients). However, these methods often introduce high variance or bias, necessitating smaller learning rates that impede training speed. Furthermore, information loss from rounding small gradients to zero can degrade final model performance.
Methodology
Quantized Compressive Sampling (QCS) addresses these challenges by treating gradient compression as a signal processing task. The method involves four key components:
- Random Mixing: Gradients are multiplied by a structured random matrix (e.g., Hadamard) to distribute information across the entire vector, making the signal more robust to subsequent compression.
- Dithered Quantization: Random noise (dither) is added before quantization to ensure the process remains unbiased.
- Estimators: The framework supports both Unbiased-QCS (preserving the expected gradient value) and MMSE-QCS (minimizing mean squared error for enhanced stability).
- Weighted Error Feedback: A residual vector tracks compression losses, which are incorporated into future updates. A "forgetting factor" ($\beta$) prevents the accumulation of stale gradient information, controlling variance.
Experimental Results
The proposed QCS method was evaluated on standard benchmarks including MNIST, CIFAR-10, and ImageNet (AlexNet).
Key results include:
- Enhanced Stability: QCS converges over a significantly wider range of learning rates compared to Sparse-SGD or QSGD, which often fail as the step size increases (see Figure 2a).
- Faster Convergence: At equivalent communication budgets, QCS-SGD reached target accuracy up to twice as fast as QSGD.
- Efficiency: On ImageNet with AlexNet, QCS achieved 100x compression with minimal computational overhead, reducing processing time per 100 batches from 55 seconds (baseline) to 22 seconds.
Discussion and Insights
The core advantage of QCS lies in its ability to transform the "spiky" noise typical of traditional compression into a distribution resembling Gaussian noise with a fixed signal-to-noise ratio. Since Stochastic Gradient Descent (SGD) is inherently robust to Gaussian noise, QCS permits the use of more aggressive hyperparameters. The use of Hadamard matrices ensures that the computational cost of mixing remains negligible.
While highly effective, QCS requires partitioning extremely large gradient layers to maintain transform efficiency and necessitates additional memory for storing residual vectors. Future research will explore the application of QCS to non-vision architectures, such as Transformer models.
Summary updated