Edge AIBench: Towards Comprehensive End-to-End Edge Computing Benchmarking

Tianshu Hao, Yunyou Huang, Xu Wen, Wanling Gao, Fan Zhang, Chen Zheng, Lei Wang, Hainan Ye, Kai Hwang, Zujie Ren, Jianfeng Zhan

TL;DR

This paper introduces Edge AIBench, a comprehensive end-to-end benchmarking suite for AI applications in edge computing. It addresses the critical need for holistic evaluation across client devices, edge servers, and cloud infrastructure, moving beyond existing benchmarks that focus on isolated layers. Edge AIBench also presents a federated learning testbed to tackle privacy and security concerns inherent in edge AI.

What Problem Does This Paper Address?

The proliferation of IoT devices and edge computing has created a demand for processing data closer to the source, reducing latency and bandwidth bottlenecks. However, evaluating the performance, privacy, and security of these complex, multi-layered systems is challenging. Existing benchmarks often focus on individual components (e.g., just the edge layer or just cloud servers), failing to capture the end-to-end performance and interactions crucial for real-world edge AI applications. This gap hinders the development and optimization of robust edge AI solutions.

Methods

Edge AIBench proposes a comprehensive, end-to-end benchmarking framework that models three distinct layers: client-side devices, the edge computing layer, and cloud servers. It defines four typical AI application scenarios: ICU Patient Monitor, Surveillance Camera, Smart Home, and Autonomous Vehicle. For each scenario, it outlines the data flow, workload distribution, and AI tasks involved across these layers, encompassing training, validation, and inference stages.

To address privacy and security concerns, the authors are also developing an edge computing testbed that integrates a federated learning framework. This approach allows models to be trained collaboratively across distributed devices without directly sharing raw data, preserving user privacy.

Datasets

The paper mentions the use of the following datasets:

  • MIMIC-III: Used for the ICU Patient Monitor scenario, providing various patient data like vital signs and fluid balance.
  • LibriSpeech: Used for the speech recognition component in the Smart Home scenario.
  • LFW (Labeled Faces in the Wild): Used for the face recognition component in the Smart Home scenario.

The paper also mentions that Edge AIBench is part of the open-source AIBench project and is publicly available.

Key Insights and Ideas

  • End-to-End is Crucial: The paper strongly emphasizes that evaluating edge AI requires a holistic, end-to-end perspective, considering the interplay between client devices, edge servers, and the cloud. Isolated layer benchmarks are insufficient.
  • Scenario-Driven Benchmarking: By modeling realistic application scenarios (ICU, surveillance, smart home, autonomous vehicles), Edge AIBench provides a more practical and representative evaluation than abstract task-based benchmarks.
  • Federated Learning for Privacy: The integration of federated learning into their testbed highlights a proactive approach to addressing the significant privacy and security challenges in edge AI, where data is distributed and sensitive.
  • Complexity of Edge AI: The chosen scenarios showcase the inherent complexity of edge AI, involving heterogeneous devices, diverse data types, and stringent performance requirements (latency, bandwidth).

Key Results

The paper primarily focuses on introducing the benchmark suite and testbed rather than presenting quantitative results from running these benchmarks. However, it states that:

  • Edge AIBench models four typical application scenarios.
  • It covers the entire edge computing AI lifecycle, including training, inference, and data collection.
  • The benchmark suite is designed to present the complexity of edge computing AI scenarios from different perspectives.
  • A federated learning testbed is being developed to resolve security and privacy issues.

Limitations and Future Work

  • Initial Stage: The paper acknowledges that the benchmark suite and testbed are in their initial stages. The testbed is currently implementing the ICU scenario.
  • Lack of Quantitative Results: The paper does not present detailed quantitative performance results from running the Edge AIBench benchmarks on specific hardware or software configurations.
  • Future Work: The authors plan to continue developing the testbed and expanding the benchmark scenarios. They also aim to make the Edge AIBench suite publicly available to foster research and development in the field.

Summary updated

Abstract

In edge computing scenarios, the distribution of data and collaboration of workloads on different layers are serious concerns for performance, privacy, and security issues. So for edge computing benchmarking, we must take an end-to-end view, considering all three layers: client-side devices, edge computing layer, and cloud servers. Unfortunately, the previous work ignores this most important point. This paper presents the BenchCouncil's coordinated e ort on edge AI benchmarks, named Edge AIBench. In total, Edge AIBench models four typical application scenarios: ICU Patient Monitor, Surveillance Camera, Smart Home, and Autonomous Vehicle with the focus on data distribution and workload collaboration on three layers. Edge AIBench is a part of the open-source AIBench project, publicly available from http://www.benchcouncil.org/AIBench/index.html. We also build an edge computing testbed with a federated learning framework to resolve performance, privacy, and security issues.
BibTeX
@inproceedings{hao2018edge,
  author = {Tianshu Hao and Yunyou Huang and Xu Wen and Wanling Gao and Fan Zhang and Chen Zheng and Lei Wang and Hainan Ye and Kai Hwang and Zujie Ren and Jianfeng Zhan},
  title = {{Edge AIBench: Towards Comprehensive End-to-End Edge Computing Benchmarking}},
  booktitle = {Bench},
  year = {2018},
  doi = {10.1007/978-3-030-32813-9\_3},
}
Figure