Quantized Compressive Sampling of Stochastic Gradients for Efficient Communication in Distributed Deep Learning

Afshin Abdi, Faramarz Fekri

Abstract

Training large-scale deep learning models is frequently limited by the communication overhead of synchronizing gradients across distributed workers. We present Quantized Compressive Sampling (QCS), a novel technique that integrates random signal mixing with dithered quantization to achieve significant gradient compression (32x or more) while preserving model accuracy. By transforming compression noise into a manageable Gaussian-like distribution, QCS enables the use of larger learning rates and achieves faster convergence compared to existing state-of-the-art quantization and sparsification methods.

Introduction

In distributed deep learning, multiple "workers" (e.g., GPUs) compute local gradients and must synchronize updates to maintain a global model. As the scale of these systems increases, the "communication bottleneck"—the time required to transmit high-dimensional gradient vectors across the network—becomes the primary performance constraint.

Distributed training architecture showing gradient exchange between workers and a central server.

Existing approaches to mitigate this bottleneck include quantization (reducing numerical precision) and sparsification (transmitting only a subset of gradients). However, these methods often introduce high variance or bias, necessitating smaller learning rates that impede training speed. Furthermore, information loss from rounding small gradients to zero can degrade final model performance.

Methodology

Quantized Compressive Sampling (QCS) addresses these challenges by treating gradient compression as a signal processing task. The method involves four key components:

  1. Random Mixing: Gradients are multiplied by a structured random matrix (e.g., Hadamard) to distribute information across the entire vector, making the signal more robust to subsequent compression.
  2. Dithered Quantization: Random noise (dither) is added before quantization to ensure the process remains unbiased.
  3. Estimators: The framework supports both Unbiased-QCS (preserving the expected gradient value) and MMSE-QCS (minimizing mean squared error for enhanced stability).
  4. Weighted Error Feedback: A residual vector tracks compression losses, which are incorporated into future updates. A "forgetting factor" ($\beta$) prevents the accumulation of stale gradient information, controlling variance.

Experimental Results

The proposed QCS method was evaluated on standard benchmarks including MNIST, CIFAR-10, and ImageNet (AlexNet).

Performance comparison of QCS against baselines: (a) convergence region for step-size $\mu$, (b) convergence rate at $\mu=0.05$, and (c) convergence rate at $\mu=0.1$.

Key results include:

  • Enhanced Stability: QCS converges over a significantly wider range of learning rates compared to Sparse-SGD or QSGD, which often fail as the step size increases (see Figure 2a).
  • Faster Convergence: At equivalent communication budgets, QCS-SGD reached target accuracy up to twice as fast as QSGD.
  • Efficiency: On ImageNet with AlexNet, QCS achieved 100x compression with minimal computational overhead, reducing processing time per 100 batches from 55 seconds (baseline) to 22 seconds.

Discussion and Insights

The core advantage of QCS lies in its ability to transform the "spiky" noise typical of traditional compression into a distribution resembling Gaussian noise with a fixed signal-to-noise ratio. Since Stochastic Gradient Descent (SGD) is inherently robust to Gaussian noise, QCS permits the use of more aggressive hyperparameters. The use of Hadamard matrices ensures that the computational cost of mixing remains negligible.

While highly effective, QCS requires partitioning extremely large gradient layers to maintain transform efficiency and necessitates additional memory for storing residual vectors. Future research will explore the application of QCS to non-vision architectures, such as Transformer models.

Summary updated

Abstract

In distributed training of deep models, the transmission volume of stochastic gradients (SG) imposes a bottleneck in scaling up the number of processing nodes. On the other hand, the existing methods for compression of SGs have two major drawbacks. First, due to the increase in the overall variance of the compressed SG, the hyperparameters of the learning algorithm must be readjusted to ensure the convergence of the training. Further, the convergence rate of the resulting algorithm still would be adversely affected. Second, for those approaches for which the compressed SG values are biased, there is no guarantee for the learning convergence and thus an error feedback is often required. We propose Quantized Compressive Sampling (QCS) of SG that addresses the above two issues while achieving an arbitrarily large compression gain. We introduce two variants of the algorithm: Unbiased-QCS and MMSE-QCS and show their superior performance w.r.t. other approaches. Specifically, we show that for the same number of communication bits, the convergence rate is improved by a factor of 2 relative to state of the art. Next, we propose to improve the convergence rate of the distributed training algorithm via a weighted error feedback. Specifically, we develop and analyze a method to both control the overall variance of the compressed SG and prevent the staleness of the updates. Finally, through simulations, we validate our theoretical results and establish the superior performance of the proposed SG compression in the distributed training of deep models. Our simulations also demonstrate that our proposed compression method expands substantially the region of step-size values for which the learning algorithm converges.
BibTeX
@inproceedings{abdi2020quantized,
  author = {Afshin Abdi and Faramarz Fekri},
  title = {{Quantized Compressive Sampling of Stochastic Gradients for Efficient Communication in Distributed Deep Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i04.5706},
}
Figure