AIBench: An Industry Standard Internet Service AI Benchmark Suite

Wanling Gao, Fei Tang, Lei Wang, Jianfeng Zhan, Chunxin Lan, Chunjie Luo, Yunyou Huang, Chen Zheng, Jiahui Dai, Zheng Cao, Daoyi Zheng, Haoning Tang, Kunlin Zhan, Biao Wang, Defei Kong, Tong Wu, Minghe Yu, Chongkang Tan, Huan Li, Xinhui Tian, Yatao Li, Junchao Shao, Zhenyu Wang, Xiaoyu Wang, Hainan YearXiv 2019 · arXiv

Introduction

Modern Internet services are increasingly reliant on Artificial Intelligence, yet benchmarking these systems has historically been fragmented. While existing suites like MLPerf or DeepBench focus on isolated components or micro-operations, they often fail to capture the complexity of real-world microservice architectures. In production environments, AI does not run in isolation; it is part of an intricate execution path where data confidentiality and massive scale introduce significant performance challenges.

AIBench, developed by BenchCouncil in collaboration with 17 industry partners, addresses this gap. It is the first industry-standard AI benchmark suite designed to represent the complex reality of modern Internet services through a hierarchical testing framework.

Comparison of AIBench with other AI benchmark suites across end-to-end, component, and micro-benchmarks.

The AIBench Framework

The authors propose a highly extensible framework organized into three layers of granularity to provide a holistic view of AI performance:

  1. Micro Benchmarks: These consist of 12 fundamental units of computation, such as Convolution, Fully Connected layers, ReLU, and Softmax, extracted from prominent AI algorithms.
  2. Component Benchmarks: This layer includes 16 prominent AI problem domains identified through industry analysis, covering tasks like Image Classification, Speech Recognition, Object Detection, and Learning to Rank.
  3. End-to-End Application Benchmark: The pinnacle of the suite is a complete implementation of an e-commerce search intelligence system. This includes an Online Server for query parsing and ranking, an Offline Analyzer for model training, and a Query Generator.
The hierarchical framework of AIBench, spanning from data input to online inference.

The framework is built for modularity, utilizing deployment tools like Ansible and Kubernetes to allow researchers to either run the entire pipeline or isolate specific modules for targeted testing.

Datasets and Data Generation

AIBench utilizes 15 representative datasets covering structured, semi-structured, and unstructured data types. These include standard vision and NLP sets like ImageNet and MS COCO, as well as specialized industry data like 3D face recognition and robot pushing datasets. To navigate data confidentiality issues common in industry, the suite provides data generators that maintain real-world characteristics while allowing for scaling across different cluster sizes.

Key Insights and Performance Analysis

The evaluation of AIBench reveals that AI components are not merely "add-ons" but fundamental drivers of data center performance.

The Impact on Latency

AI components significantly impact both average and tail latency. In the e-commerce testbed, the "Recommender" module (using TensorFlow Serving) was identified as a primary bottleneck. While its average response time was approximately 75ms, its 99th percentile latency spiked to over 550ms. This "long tail" effect is a critical challenge for production AI services.

Hardware Efficiency and Bottlenecks

The suite highlights a "GPU Efficiency Paradox." While some tasks like Recommendation achieve high Streaming Multiprocessor (SM) efficiency (95%), others like Learning to Rank hover around 29%. Detailed stall analysis indicates that memory dependencies are the leading cause of GPU idle time, accounting for up to 68% of stalls in certain kernels. This suggests that future optimizations should prioritize data alignment and locality over raw compute power.

Kernel Hotspots

Across all component benchmarks, six specific kernels—Convolution, GEMM, BatchNorm, ReLU, Element-wise, and Gradient—occupy over 80% of total running time. These "hotspots" provide a clear roadmap for systems engineers looking to optimize AI workloads.

Conclusion and Future Directions

AIBench provides a standardized lens through which to view AI in the wild, bridging the gap between standalone model accuracy and production-level system performance. While the current baseline uses general-purpose frameworks like Flask, which may have higher latency than proprietary industry solutions, it provides a vital open standard for research.

Future work involves expanding the suite to other verticals, such as clinical diagnosis in "Medicine AI," and refining data generators to better simulate the "data drift" encountered in live production environments.

Summary updated

Abstract

Today's Internet Services are undergoing fundamental changes and shifting to an intelligent computing era where AI is widely employed to augment services. In this context, many innovative AI algorithms, systems, and architectures are proposed, and thus the importance of benchmarking and evaluating them rises. However, modern Internet services adopt a microservice-based architecture and consist of various modules. The diversity of these modules and complexity of execution paths, the massive scale and complex hierarchy of datacenter infrastructure, the confidential issues of data sets and workloads pose great challenges to benchmarking. In this paper, we present the first industry-standard Internet service AI benchmark suite---AIBench with seventeen industry partners, including several top Internet service providers. AIBench provides a highly extensible, configurable, and flexible benchmark framework that contains loosely coupled modules. We identify sixteen prominent AI problem domains like learning to rank, each of which forms an AI component benchmark, from three most important Internet service domains: search engine, social network, and e-commerce, which is by far the most comprehensive AI benchmarking effort. On the basis of the AIBench framework, abstracting the real-world data sets and workloads from one of the top e-commerce providers, we design and implement the first end-to-end Internet service AI benchmark, which contains the primary modules in the critical paths of an industry scale application and is scalable to deploy on different cluster scales. The specifications, source code, and performance numbers are publicly available from the benchmark council web site http://www.benchcouncil.org/AIBench/index.html.
BibTeX
@misc{gao2019aibench,
  author = {Wanling Gao and Fei Tang and Lei Wang and Jianfeng Zhan and Chunxin Lan and Chunjie Luo and Yunyou Huang and Chen Zheng and Jiahui Dai and Zheng Cao and Daoyi Zheng and Haoning Tang and Kunlin Zhan and Biao Wang and Defei Kong and Tong Wu and Minghe Yu and Chongkang Tan and Huan Li and Xinhui Tian and Yatao Li and Junchao Shao and Zhenyu Wang and Xiaoyu Wang and Hainan Ye},
  title = {{AIBench: An Industry Standard Internet Service AI Benchmark Suite}},
  year = {2019},
  eprint = {1908.08998},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.08998},
}
Figure