Musical Chair: Efficient Real-Time Recognition Using Collaborative IoT Devices

Ramyad Hadidi, Jiashen Cao, Matthew Woodward, Michael S. Ryoo, Hyesoon KimarXiv 2018 · arXiv

Abstract

Cloud-based deep learning for IoT devices introduces significant privacy risks and latency, yet individual edge devices often lack the computational resources required for real-time video and image recognition. This paper presents "Musical Chair," a distributed inference system that pools the resources of multiple low-power IoT devices (e.g., Raspberry Pis) using dynamic data and model parallelism. By intelligently distributing neural network layers across a local network, Musical Chair achieves real-time performance that rivals or exceeds dedicated edge accelerators like the Nvidia Jetson TX2.

Introduction

As IoT devices such as smart cameras and sensors proliferate, they generate massive volumes of raw data. Processing this data with Deep Neural Networks (DNNs) for tasks like action or image recognition is computationally intensive. The industry standard involves offloading this processing to the cloud, which consumes significant bandwidth, introduces latency, and raises privacy concerns.

Comparison of inference architectures: (a) External Server, (b) In-Home Server, and (c) the proposed Musical Chair distributed system.

Performing inference locally at the edge is the ideal alternative. However, a single resource-constrained IoT device typically lacks the memory and processing power to run state-of-the-art DNNs in real-time. When a single device attempts to run a massive model, it spends the majority of its time swapping weights in and out of memory, causing performance to stall. Musical Chair addresses this by harvesting the aggregated power of multiple devices on a local network.

The Musical Chair Approach

Musical Chair utilizes a distributed inference system that combines two primary strategies:

  • Data Parallelism: Tasks are duplicated across multiple devices to process independent inputs concurrently.
  • Model Parallelism: Computationally heavy layers are split across multiple devices to reduce the memory and compute burden on any single node.

System Workflow

The system first profiles target DNN layers to understand memory usage and cache miss ratios. Based on this profiling and available network overhead, an algorithm assigns specific layers to specific devices. To handle real-time video, the system employs a "sliding window" approach, buffering incoming frames and optical flow data to ensure distributed layers have continuous inputs.

The two-stream action recognition model architecture distributed by Musical Chair.

Communication is managed via Apache Avro for lightweight RPC, allowing the system to dynamically update task assignments—effectively playing a game of "musical chairs"—if the environment or device availability changes.

Key Insights

Research into the system revealed that different DNN layers require different parallelism strategies. Fully Connected (FC) layers are heavily memory-bound with high cache miss ratios; they benefit most from model parallelism to avoid memory swapping. Conversely, Convolutional (Conv) layers are compute-bound but highly reusable, making data parallelism the superior choice.

By eliminating the setup and reloading time associated with memory swapping on a single device, Musical Chair unlocks massive speedups. This demonstrates that collective edge compute can rival specialized, expensive hardware by networking cheap, existing IoT devices.

Evaluation and Results

The system was evaluated using up to 12 Raspberry Pi 3s connected via local WiFi, compared against a high-performance Titan Xp GPU and an Nvidia Jetson TX2.

  • Action Recognition: A 12-device Raspberry Pi setup achieved 10.00 Inferences Per Second (IPS), nearly doubling the performance of the Jetson TX2's GPU (5.34 IPS).
  • Energy Efficiency: Compared to a single Raspberry Pi, the distributed setup achieved a 90x energy savings and a 500x speedup. It also proved more energy-efficient per inference than the HPC machine.
  • Image Recognition: For the AlexNet model, a 6-device system matched the TX2-CPU. For the heavier VGG16 model, an 11-device system achieved performance comparable to the TX2-GPU, with only a 15% degradation.

Conclusion and Future Work

While Musical Chair demonstrates the power of collaborative edge computing, it faces limitations regarding static energy overhead and network dependency. Raspberry Pis lack the advanced power-gating of specialized AI boards, and performance is sensitive to local WiFi congestion. Future work will explore combining Musical Chair with edge-optimized libraries (like TensorFlow Lite) and model compression techniques such as pruning and quantization to further enhance efficiency.

Summary updated

Abstract

The prevalence of Internet of things (IoT) devices and abundance of sensor data has created an increase in real-time data processing such as recognition of speech, image, and video. While currently such processes are offloaded to the computationally powerful cloud system, a localized and distributed approach is desirable because (i) it preserves the privacy of users and (ii) it omits the dependency on cloud services. However, IoT networks are usually composed of resource-constrained devices, and a single device is not powerful enough to process real-time data. To overcome this challenge, we examine data and model parallelism for such devices in the context of deep neural networks. We propose Musical Chair to enable efficient, localized, and dynamic real-time recognition by harvesting the aggregated computational power from the resource-constrained devices in the same IoT network as input sensors. Musical chair adapts to the availability of computing devices at runtime and adjusts to the inherit dynamics of IoT networks. To demonstrate Musical Chair, on a network of Raspberry PIs (up to 12) each connected to a camera, we implement a state-of-the-art action recognition model for videos and two recognition models for images. Compared to the Tegra TX2, an embedded low-power platform with a six-core CPU and a GPU, our distributed action recognition system achieves not only similar energy consumption but also twice the performance of the TX2. Furthermore, in image recognition, Musical Chair achieves similar performance and saves dynamic energy.
BibTeX
@misc{hadidi2018musical,
  author = {Ramyad Hadidi and Jiashen Cao and Matthew Woodward and Michael S. Ryoo and Hyesoon Kim},
  title = {{Musical Chair: Efficient Real-Time Recognition Using Collaborative IoT Devices}},
  year = {2018},
  eprint = {1802.02138},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.02138},
}
Figure