Abstract
We consider machine learning applications that train a model by leveraging data distributed over a trusted network, where communication constraints can create a performance bottleneck. In this paper, we propose an alternate approach to learn from distributed data that quantizes data instead of gradients, and can support learning over applications where the size of gradient updates is prohibitive. Our approach leverages the dependency of the computed gradient on data samples, which lie in a much smaller space in order to perform the quantization in the smaller dimension data space. We analyze the convergence of the proposed approach for smooth convex and non-convex objective functions and show that we can achieve order optimal convergence rates with communication that mostly depends on the data rather than the model (gradient) dimension. We show that we can achieve an order of magnitude savings over gradient compression methods at the cost of increasing computation at the learning agent.
The Communication Bottleneck and the Dimension Gap
In distributed machine learning, a central server (the agent) trains a model using data streamed from various nodes, such as sensors or mobile devices. Traditionally, the "communication bottleneck" is addressed through gradient compression—shrinking the updates the nodes send back to the server.
However, as we enter an era of "model explosion," neural networks (with model dimension $h$) are growing to millions or billions of parameters, while the input data (with dimension $d$) remains relatively constant. If $h \gg d$, quantizing the gradient is inherently less efficient than quantizing the data. The authors propose DaQuSGD (Dataset-Quantized Stochastic Gradient Descent), which shifts the focus from compressing the model's output to compressing its input.
Methodology: The DaQuSGD Framework
DaQuSGD consists of three main pillars designed to minimize communication while maintaining model accuracy:
- DataQ (Datapoint Quantization): This algorithm maps high-dimensional data points into a discrete, low-precision space. It uses a "splitting" technique to represent signs implicitly, saving significant bits during transmission.
- GradCorrQ (Gradient Correction): Quantizing data can introduce bias. To resolve this, nodes can send a single stochastic bit to the server, allowing the agent to "correct" the gradient estimate. This ensures the model remains theoretically guaranteed to converge at the optimal $O(1/\sqrt{n})$ rate.
- Sample Selection (Thresholding): Not all data is equally useful for training. Borrowing intuition from Support Vector Machines (SVMs), nodes perform a local "forward pass" to calculate a sample's loss. If the loss is below a certain threshold—meaning the model already "knows" this point—the node does not transmit it.
As illustrated in the table above, DaQuSGD achieves order-optimal convergence rates while decoupling communication costs from the model dimension $h$, shifting the computational load of gradient calculation to the learner.
Key Results and Performance
The researchers evaluated DaQuSGD on standard computer vision benchmarks, training ResNet-18 on CIFAR-10 and ResNet-50 on ImageNet.
- Massive Communication Savings: On CIFAR-10, DaQuSGD achieved a 14x reduction in bits compared to QTopK-SGD and a 300x reduction compared to QSGD for the same target accuracy. On the larger ImageNet dataset, the method offered a 7x saving over QTopK-SGD.
- Accuracy Preservation: Despite heavy quantization, final top-1 accuracies remained comparable to unquantized SGD, reaching approximately 93% on CIFAR-10 and 73.6% on ImageNet.
- Theoretical Convergence: The analysis confirms that the model converges at the same "order-optimal" rate as standard SGD, meaning the communication savings do not require a significant increase in training iterations.
Limitations and Future Work
The primary trade-off of DaQuSGD is the increased server computation, as the learning agent must compute gradients that were previously handled by nodes. This makes the method ideal for scenarios with powerful servers but weak network links. Future research will focus on integrating Differential Privacy for sensitive data and automating the tuning of sample selection thresholds.
Summary updated