JointSQ: Joint Sparsification-Quantization for Distributed Learning

Weiying Xie, Haowei Li, Jitao Ma, Yunsong Li, Jie Lei, Donglai Liu, Leyuan Fang

Introduction

In distributed learning, the communication of gradient updates between worker nodes and a central server often becomes a significant bottleneck, particularly in bandwidth-constrained environments like edge computing. To mitigate this, researchers have traditionally relied on two distinct techniques: sparsification (transmitting only the most significant gradients) and quantization (reducing the bit-precision of transmitted values).

Conventionally, these methods are applied sequentially in a "two-stage" pipeline. However, this approach is inherently suboptimal because the stages are decoupled; the sparsification step is unaware of the subsequent quantization requirements, leading to accumulated noise and inefficient use of the communication budget. JointSQ addresses this by unifying these processes into a single, optimized framework.

Conceptual comparison between traditional sequential co-compression and the proposed JointSQ framework.

The JointSQ Framework

The fundamental insight of JointSQ is that sparsification can be mathematically treated as a special case of quantization—specifically, "0-bit quantization." By adopting this perspective, the problem of gradient compression is transformed into a mixed-precision optimization task.

1. Mixed-Precision Formulation

Instead of applying a uniform bit-width across all gradients, JointSQ allows each gradient element to be assigned a bit-width from a predefined candidate set (e.g., {0, 2, 4, 8} bits). This flexibility allows the system to prioritize precision for the most informative gradients while effectively "sparsifying" less critical ones by assigning them 0 bits.

2. Optimization via the Knapsack Problem

The authors frame the bit-assignment task as a Multiple-Choice Knapsack Problem (MCKP). In this model, the "profit" is defined as the reduction in compression noise (error) achieved by increasing bit-precision, while the "weight" corresponds to the number of bits used. The objective is to minimize total noise subject to a strict communication bandwidth constraint.

3. Greedy Allocation and Reallocation

To solve the NP-hard MCKP efficiently during training, JointSQ employs a two-step heuristic:

  • Greedy Allocation: Gradients with larger magnitudes are prioritized for higher bit-widths, as they typically carry more critical information for model convergence.
  • Reallocation: An iterative fine-tuning step adjusts bit assignments across different layers to further minimize global noise while adhering to the total bit budget.
The JointSQ workflow, illustrating the greedy allocation and reallocation process for mixed-precision quantization.

Experimental Evaluation

JointSQ was evaluated across a variety of tasks, including image classification (CIFAR-10/100, ImageNet) and language modeling (Penn Treebank), using architectures ranging from CNNs (ResNet, VGG) to Transformers (SimpleViT) and RNNs.

Key Results

  • Extreme Compression: JointSQ achieved up to a 1000× compression ratio on ResNet and VGG models with less than a 1% drop in Top-1 accuracy.
  • Training Speedup: In multi-node distributed environments, the framework provided 1.4× to 2.9× speedups in total training time compared to state-of-the-art sequential co-compression methods.
  • Robustness: Vision Transformers, which are typically sensitive to gradient noise, maintained near-baseline accuracy at 20× compression. Interestingly, on ImageNet, JointSQ occasionally outperformed the baseline SGD, suggesting that quantization noise may provide a beneficial regularizing effect.

Discussion and Future Work

While JointSQ demonstrates superior performance, the authors identified several avenues for future research. High compression ratios at the very beginning of training can occasionally hinder initial convergence. Preliminary results suggest that employing "periodic variable compression ratios"—starting with lower compression and increasing it over time—could further optimize the training trajectory. Additionally, while the algorithm is computationally efficient, full-scale deployment would benefit from specialized hardware kernels designed to handle mixed-precision gradients natively on GPUs.

Summary updated

Abstract

Gradient sparsification and quantization offer a promising prospect to alleviate the communication overhead problem in distributed learning. However, direct combination of the two results in suboptimal solutions, due to the fact that sparsification and quantization haven't been learned together. In this paper, we propose Joint Sparsification-Quantization (JointSQ) inspired by the discovery that sparsification can be treated as 0-bit quantization, regardless of architectures. Specifically, we mathematically formu-late JointSQ as a mixed-precision quantization problem, expanding the solution space. It can be solved by the designed MCKP-Greedy algorithm. Theoretical analysis demon-strates the minimal compression noise of JointSQ, and ex-tensive experiments on various network architectures, including CNN, RNN, and Transformer, also validate this point. Under the introduction of computation overhead consistent with or even lower than previous methods, JointSQ achieves a compression ratio of 1000× on different models while maintaining near-lossless accuracy and brings 1.4× to 2.9× speedup over existing methods.
BibTeX
@inproceedings{xie2024jointsq,
  author = {Weiying Xie and Haowei Li and Jitao Ma and Yunsong Li and Jie Lei and Donglai Liu and Leyuan Fang},
  title = {{JointSQ: Joint Sparsification-Quantization for Distributed Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00552},
}
Figure