Scalable Smartphone Cluster for Deep Learning

Byunggook Na, Jaehee Jang, Seongsik Park, Sei Joon Kim, Joonoo Kim, Moon-Sik Jeong, Kwang-Choon Kim, Seon Heo, Yoonsang Kim, Sungroh YoonarXiv 2021 · OpenReview

TL;DR

This paper presents a novel approach to deep learning training by building a large-scale cluster of 138 smartphones connected via Ethernet. By moving away from portable, wireless clusters and employing large-batch training, they demonstrate that smartphone clusters can achieve significant computational efficiency, rivaling high-end GPUs for certain tasks.

What Problem Does This Paper Address?

Training deep neural networks (DNNs) is computationally intensive, far exceeding the capabilities of a single smartphone. While portable clusters of smartphones have been explored for parallel computation, they are typically limited by wireless connectivity to small scales (under 30 devices) and lack the power to train complex DNNs from scratch. This paper addresses the challenge of building a scalable and powerful smartphone cluster capable of deep learning training, aiming to provide a more accessible and environmentally friendly alternative to traditional GPU clusters.

Methods

The core of the paper is the construction and evaluation of a large-scale smartphone cluster. Key methodological aspects include:

  • Hardware Setup: A cluster of 138 Samsung Galaxy S10+ smartphones was assembled. Crucially, they opted for a wired Ethernet connection instead of wireless (like Wi-Fi) to overcome the limitations of bandwidth and stability. Multi-port adaptors were used to provide both power and data connectivity to each smartphone.
  • Software Framework: The cluster utilizes the Caffe deep learning library with OpenCL for leveraging the smartphone's processing capabilities (AP-CPUs and AP-GPU). OpenMPI was used for inter-smartphone communication.
  • Distributed Training Strategy: They implemented synchronous training based on data parallelism. A key innovation is the adoption of large-batch training techniques. This involves maximizing the mini-batch size per device to fully utilize the available memory, thereby reducing the total number of training iterations needed.
  • Gradient Aggregation: To improve communication efficiency, they implemented gradient packing, where gradients from all layers are aggregated into a single chunk before transmission, reducing the overhead of multiple communication calls. They compared this to a "chunk-wise" transfer and found gradient packing to be more efficient for deeper networks.

Datasets

The paper evaluates their system on various standard deep learning models trained on datasets like ImageNet and CIFAR10. Specific models mentioned include AlexNet, GoogleNet, Inception-v3, MobileNet-v1, MobileNet-v2, ResNet-50, ResNet-101, ResNet-152, and SqueezeNet (v1.0 and v1.1). No new datasets were introduced.

Key Insights and Ideas

  • Wired vs. Wireless is Crucial for Scale: The paper strongly demonstrates that for large-scale smartphone clusters, wired Ethernet is not just preferable but essential. Wireless connections quickly become bottlenecks and unstable beyond a small number of devices, severely limiting scalability.
  • Large-Batch Training is Key to Smartphone Cluster Efficiency: Simply scaling up data parallelism with small batches doesn't work well. By adopting large-batch training, they can maximize the utilization of each smartphone's limited memory, leading to significant performance gains.
  • Repurposing Obsolete Smartphones is Environmentally Sound: The authors highlight the environmental benefit of using discarded smartphones, turning potential e-waste into a valuable computational resource.
  • Smartphone Clusters Can Be Competitive: While not always outperforming top-tier GPUs, the cluster shows remarkable performance, especially for mobile-optimized networks like MobileNet, suggesting a viable alternative for certain deep learning tasks.

Key Results

  • The smartphone cluster achieved 90% of the speed of a P100 GPU when training ResNet-50.
  • For MobileNet-v1 training, the cluster achieved an impressive 43x speed-up compared to a V100 GPU.
  • The cluster demonstrated comparable performance to GPGPUs for training various DNNs, with notable strengths in MobileNet models.
  • They achieved 85.8% computational efficiency on SqueezeNet-v1.1, though efficiency dropped for larger models like ResNet-152 due to memory limitations.
  • The wired Ethernet connection supported 138 smartphones stably, with speeds more than twice as fast and more stable than Wi-Fi-5GHz.

Limitations and Future Work

  • Thermal Throttling: Smartphones, unlike servers, lack robust cooling systems. The paper notes that without active cooling (fans), temperatures rise, leading to thermal throttling and reduced performance. Future work will require more elaborate cooling solutions.
  • Memory Limitations: For very large models like ResNet-152, the limited memory on each smartphone restricts the mini-batch size per device, impacting computational efficiency and scaling.
  • Heterogeneous Clusters: The current cluster uses identical Galaxy S10+ devices. Future work aims to build clusters with heterogeneous devices, which will require adapting distributed training strategies to handle varying computational capabilities.
  • Scaling to Thousands: The authors aspire to build clusters with thousands of smartphones, which will necessitate further research into network topologies, collective communication primitives, and stable training without performance degradation.

Summary updated

Abstract

Various deep learning applications on smartphones have been rapidly rising, but training deep neural networks (DNNs) has too large computational burden to be executed on a single smartphone. A portable cluster, which connects smartphones with a wireless network and supports parallel computation using them, can be a potential approach to resolve the issue. However, by our findings, the limitations of wireless communication restrict the cluster size to up to 30 smartphones. Such small-scale clusters have insufficient computational power to train DNNs from scratch. In this paper, we propose a scalable smartphone cluster enabling deep learning training by removing the portability to increase its computational efficiency. The cluster connects 138 Galaxy S10+ devices with a wired network using Ethernet. We implemented large-batch synchronous training of DNNs based on Caffe, a deep learning library. The smartphone cluster yielded 90% of the speed of a P100 when training ResNet-50, and approximately 43x speed-up of a V100 when training MobileNet-v1.
BibTeX
@misc{na2021scalable,
  author = {Byunggook Na and Jaehee Jang and Seongsik Park and Sei Joon Kim and Joonoo Kim and Moon-Sik Jeong and Kwang-Choon Kim and Seon Heo and Yoonsang Kim and Sungroh Yoon},
  title = {{Scalable Smartphone Cluster for Deep Learning}},
  year = {2021},
  eprint = {2110.12172},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.12172},
}
Figure