Quantizing data for distributed learning

Osama A. Hanna, Yahya H. Ezzeldin, Christina Fragouli, Suhas N. Diggavi

Abstract

We consider machine learning applications that train a model by leveraging data distributed over a trusted network, where communication constraints can create a performance bottleneck. In this paper, we propose an alternate approach to learn from distributed data that quantizes data instead of gradients, and can support learning over applications where the size of gradient updates is prohibitive. Our approach leverages the dependency of the computed gradient on data samples, which lie in a much smaller space in order to perform the quantization in the smaller dimension data space. We analyze the convergence of the proposed approach for smooth convex and non-convex objective functions and show that we can achieve order optimal convergence rates with communication that mostly depends on the data rather than the model (gradient) dimension. We show that we can achieve an order of magnitude savings over gradient compression methods at the cost of increasing computation at the learning agent.

The Communication Bottleneck and the Dimension Gap

In distributed machine learning, a central server (the agent) trains a model using data streamed from various nodes, such as sensors or mobile devices. Traditionally, the "communication bottleneck" is addressed through gradient compression—shrinking the updates the nodes send back to the server.

However, as we enter an era of "model explosion," neural networks (with model dimension $h$) are growing to millions or billions of parameters, while the input data (with dimension $d$) remains relatively constant. If $h \gg d$, quantizing the gradient is inherently less efficient than quantizing the data. The authors propose DaQuSGD (Dataset-Quantized Stochastic Gradient Descent), which shifts the focus from compressing the model's output to compressing its input.

Methodology: The DaQuSGD Framework

DaQuSGD consists of three main pillars designed to minimize communication while maintaining model accuracy:

  1. DataQ (Datapoint Quantization): This algorithm maps high-dimensional data points into a discrete, low-precision space. It uses a "splitting" technique to represent signs implicitly, saving significant bits during transmission.
  2. GradCorrQ (Gradient Correction): Quantizing data can introduce bias. To resolve this, nodes can send a single stochastic bit to the server, allowing the agent to "correct" the gradient estimate. This ensures the model remains theoretically guaranteed to converge at the optimal $O(1/\sqrt{n})$ rate.
  3. Sample Selection (Thresholding): Not all data is equally useful for training. Borrowing intuition from Support Vector Machines (SVMs), nodes perform a local "forward pass" to calculate a sample's loss. If the loss is below a certain threshold—meaning the model already "knows" this point—the node does not transmit it.
Comparison of DaQuSGD variants against standard gradient compression methods.

As illustrated in the table above, DaQuSGD achieves order-optimal convergence rates while decoupling communication costs from the model dimension $h$, shifting the computational load of gradient calculation to the learner.

Visualization of sample selection regions based on model loss, where P2 represents the critical region for learning.

Key Results and Performance

The researchers evaluated DaQuSGD on standard computer vision benchmarks, training ResNet-18 on CIFAR-10 and ResNet-50 on ImageNet.

  • Massive Communication Savings: On CIFAR-10, DaQuSGD achieved a 14x reduction in bits compared to QTopK-SGD and a 300x reduction compared to QSGD for the same target accuracy. On the larger ImageNet dataset, the method offered a 7x saving over QTopK-SGD.
  • Accuracy Preservation: Despite heavy quantization, final top-1 accuracies remained comparable to unquantized SGD, reaching approximately 93% on CIFAR-10 and 73.6% on ImageNet.
  • Theoretical Convergence: The analysis confirms that the model converges at the same "order-optimal" rate as standard SGD, meaning the communication savings do not require a significant increase in training iterations.

Limitations and Future Work

The primary trade-off of DaQuSGD is the increased server computation, as the learning agent must compute gradients that were previously handled by nodes. This makes the method ideal for scenarios with powerful servers but weak network links. Future research will focus on integrating Differential Privacy for sensitive data and automating the tuning of sample selection thresholds.

Summary updated

Abstract

We consider machine learning applications that train a model by leveraging data distributed over a trusted network, where communication constraints can create a performance bottleneck. A number of recent approaches propose to overcome this bottleneck through compression of gradient updates. However, as models become larger, so does the size of the gradient updates. In this paper, we propose an alternate approach to learn from distributed data that quantizes data instead of gradients, and can support learning over applications where the size of gradient updates is prohibitive. Our approach leverages the dependency of the computed gradient on data samples, which lie in a much smaller space in order to perform the quantization in the smaller dimension data space. At the cost of an extra gradient computation, the gradient estimate can be refined by conveying the difference between the gradient at the quantized data point and the original gradient using a small number of bits. Lastly, in order to save communication, our approach adds a layer that decides whether to transmit a quantized data sample or not based on its importance for learning. We analyze the convergence of the proposed approach for smooth convex and non-convex objective functions and show that we can achieve order optimal convergence rates with communication that mostly depends on the data rather than the model (gradient) dimension. We use our proposed algorithm to train ResNet models on the CIFAR-10 and ImageNet datasets, and show that we can achieve an order of magnitude savings over gradient compression methods. These communication savings come at the cost of increasing computation at the learning agent, and thus our approach is beneficial in scenarios where communication load is the main problem.
BibTeX
@article{hanna2021quantizing,
  author = {Osama A. Hanna and Yahya H. Ezzeldin and Christina Fragouli and Suhas N. Diggavi},
  title = {{Quantizing data for distributed learning}},
  journal = {IEEE J. Sel. Areas Inf. Theory},
  year = {2021},
  url = {https://openreview.net/forum?id=JW4aEpHlDL},
}
Figure