Distributed deep learning on edge-devices: feasibility via adaptive compression

Corentin Hardy, Erwan Le Merrer, Bruno Sericola

Abstract

A large portion of data mining and analytic services use modern machine learning techniques, such as deep learning. The state-of-the-art results by deep learning come at the price of an intensive use of computing resources. The leading frameworks (e.g., TensorFlow) are executed on GPUs or on high-end servers in datacenters. On the other end, there is a proliferation of personal devices with possibly free CPU cycles; this can enable services to run in users' homes, embedding machine learning operations. In this paper, we ask the following question: Is distributed deep learning computation on WAN connected devices feasible, in spite of the traffic caused by learning tasks? We show that such a setup rises some important challenges, most notably the ingress traffic that the servers hosting the up-to-date model have to sustain. In order to reduce this stress, we propose adaComp, a novel algorithm for compressing worker updates to the model on the server. Applicable to stochastic gradient descent based approaches, it combines efficient gradient selection and learning rate modulation. We then experiment and measure the impact of compression, device heterogeneity and reliability on the accuracy of learned models, with an emulator platform that embeds TensorFlow into Linux containers. We report a reduction of the total amount of data sent by workers to the server by two order of magnitude (e.g., 191-fold reduction for a convolutional network on the MNIST dataset), when compared to a standard asynchronous stochastic gradient descent, while preserving model accuracy.

Introduction

Training deep neural networks (DNNs) typically requires massive compute power found in centralized datacenters. While there is a surplus of "free" CPU cycles on billions of edge devices (IoT, PCs, routers), using them for distributed training is notoriously difficult. The primary bottleneck is the Parameter Server (PS) architecture. Over a Wide Area Network (WAN), the sheer volume of data being pushed by hundreds of workers can overwhelm the server's inbound bandwidth. Furthermore, because workers operate asynchronously, their updates are often "stale"—calculated on an old version of the model—which can destabilize training.

The AdaComp Algorithm

To address these challenges, we propose AdaComp (Adaptive Compression), a novel algorithm that attacks the bandwidth problem from two angles:

  1. Gradient Selection: Instead of sending all calculated gradients, workers only send the top $c\%$ largest updates (e.g., the top 1% or 10%) for every weight matrix and bias vector. This exploits the natural sparsity of neural network updates.
  2. Per-Parameter Adaptive Learning Rate: Standard methods handle "staleness" by lowering the learning rate for a whole update if it arrives late. AdaComp tracks staleness individually for every single parameter ($\sigma_k$). If a specific parameter has been updated many times by other workers since a device last checked in, the learning rate for that specific parameter is reduced.
The Parameter Server (PS) architecture used in AdaComp, illustrating the asynchronous push/pull mechanism and the per-parameter update logic.

Experimental Evaluation

The researchers evaluated this using an emulation platform with 200 workers running in Linux containers (LXC) to precisely measure real-world TCP traffic. The study primarily uses the MNIST dataset to simulate a complex distributed environment with a Convolutional Neural Network (CNN) and a Multi-Layer Perceptron (MLP).

Key Results

AdaComp demonstrates that sparsity is a powerful tool for distributed learning. By discarding up to 99% of gradient data and compensating with granular staleness tracking, the system achieves massive traffic savings without sacrificing accuracy.

Performance comparison of AdaComp against standard ASGD and Comp-ASGD. The left plot shows convergence over iterations, while the right plot highlights the dramatic reduction in ingress traffic required to reach high accuracy.
  • Massive Traffic Savings: To reach a 97% accuracy threshold on a CNN, standard asynchronous SGD (ASGD) required nearly 250GB of ingress traffic. AdaComp reached the same goal with just 1.33GB—a 191-fold reduction.
  • High Accuracy: AdaComp achieved a final accuracy of 98.7% on MNIST, outperforming both standard ASGD (97.85%) and simpler compression methods.
  • Resilience and Heterogeneity: The system proved remarkably robust to device failures (churn). Even when 50% of the workers crashed, the final accuracy dropped by less than 0.3%. It also successfully managed a mix of "fast" and "slow" devices, with fast devices driving early progress and slow devices contributing to final fine-tuning.

Conclusion

AdaComp makes distributed deep learning on WAN-connected edge devices feasible by significantly reducing the ingress traffic at the Parameter Server. While it increases computational load on the server to track individual parameter versions, this is a worthy trade-off for limited WAN bandwidth. Future work includes testing on larger datasets like ImageNet and adapting popular optimizers like Adam to the AdaComp framework.

Summary updated

Abstract

A large portion of data mining and analytic services use modern machine learning techniques, such as deep learning. The state-of-the-art results by deep learning come at the price of an intensive use of computing resources. The leading frameworks (e.g., TensorFlow) are executed on GPUs or on high-end servers in datacenters. On the other end, there is a proliferation of personal devices with possibly free CPU cycles; this can enable services to run in users' homes, embedding machine learning operations. In this paper, we ask the following question: Is distributed deep learning computation on WAN connected devices feasible, in spite of the traffic caused by learning tasks? We show that such a setup rises some important challenges, most notably the ingress traffic that the servers hosting the up-to-date model have to sustain. In order to reduce this stress, we propose adaComp, a novel algorithm for compressing worker updates to the model on the server. Applicable to stochastic gradient descent based approaches, it combines efficient gradient selection and learning rate modulation. We then experiment and measure the impact of compression, device heterogeneity and reliability on the accuracy of learned models, with an emulator platform that embeds TensorFlow into Linux containers. We report a reduction of the total amount of data sent by workers to the server by two order of magnitude (e.g., 191-fold reduction for a convolutional network on the MNIST dataset), when compared to a standard asynchronous stochastic gradient descent, while preserving model accuracy.
BibTeX
@inproceedings{hardy2017distributed,
  author = {Corentin Hardy and Erwan Le Merrer and Bruno Sericola},
  title = {{Distributed deep learning on edge-devices: feasibility via adaptive compression}},
  booktitle = {NCA},
  year = {2017},
  url = {https://openreview.net/forum?id=DSQKOAuhrD6},
}
Figure