Respipe: Resilient Model-Distributed DNN Training at Edge Networks

Pengzhen Li, Erdem Koyuncu, Hulya SeferogluICASSP 2021 · OpenReview

Abstract

The traditional approach to distributed deep neural network (DNN) training is data-distributed learning, which partitions and distributes data to workers. This approach, although has good convergence properties, has high communication cost, which puts a strain especially on edge systems and increases delay. An emerging approach is model-distributed learning, where a training model is distributed across workers. Model-distributed learning is a promising approach to reduce communication and storage costs, which is crucial for edge systems. In this paper, we design ResPipe, a novel resilient model-distributed DNN training mechanism against delayed/failed workers. We analyze the communication cost of ResPipe and demonstrate the trade-off between resiliency and communication cost. We implement ResPipe in a real testbed consisting of Android-based smartphones, and show that it improves the convergence rate and accuracy of training for convolutional neural networks (CNNs).
BibTeX
@inproceedings{li2021respipe,
  author = {Pengzhen Li and Erdem Koyuncu and Hulya Seferoglu},
  title = {{Respipe: Resilient Model-Distributed DNN Training at Edge Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=CTxnMaj43V},
}

Loading related content...

Figure