Introduction
Modern Internet services are increasingly reliant on Artificial Intelligence, yet benchmarking these systems has historically been fragmented. While existing suites like MLPerf or DeepBench focus on isolated components or micro-operations, they often fail to capture the complexity of real-world microservice architectures. In production environments, AI does not run in isolation; it is part of an intricate execution path where data confidentiality and massive scale introduce significant performance challenges.
AIBench, developed by BenchCouncil in collaboration with 17 industry partners, addresses this gap. It is the first industry-standard AI benchmark suite designed to represent the complex reality of modern Internet services through a hierarchical testing framework.
The AIBench Framework
The authors propose a highly extensible framework organized into three layers of granularity to provide a holistic view of AI performance:
- Micro Benchmarks: These consist of 12 fundamental units of computation, such as Convolution, Fully Connected layers, ReLU, and Softmax, extracted from prominent AI algorithms.
- Component Benchmarks: This layer includes 16 prominent AI problem domains identified through industry analysis, covering tasks like Image Classification, Speech Recognition, Object Detection, and Learning to Rank.
- End-to-End Application Benchmark: The pinnacle of the suite is a complete implementation of an e-commerce search intelligence system. This includes an Online Server for query parsing and ranking, an Offline Analyzer for model training, and a Query Generator.
The framework is built for modularity, utilizing deployment tools like Ansible and Kubernetes to allow researchers to either run the entire pipeline or isolate specific modules for targeted testing.
Datasets and Data Generation
AIBench utilizes 15 representative datasets covering structured, semi-structured, and unstructured data types. These include standard vision and NLP sets like ImageNet and MS COCO, as well as specialized industry data like 3D face recognition and robot pushing datasets. To navigate data confidentiality issues common in industry, the suite provides data generators that maintain real-world characteristics while allowing for scaling across different cluster sizes.
Key Insights and Performance Analysis
The evaluation of AIBench reveals that AI components are not merely "add-ons" but fundamental drivers of data center performance.
The Impact on Latency
AI components significantly impact both average and tail latency. In the e-commerce testbed, the "Recommender" module (using TensorFlow Serving) was identified as a primary bottleneck. While its average response time was approximately 75ms, its 99th percentile latency spiked to over 550ms. This "long tail" effect is a critical challenge for production AI services.
Hardware Efficiency and Bottlenecks
The suite highlights a "GPU Efficiency Paradox." While some tasks like Recommendation achieve high Streaming Multiprocessor (SM) efficiency (95%), others like Learning to Rank hover around 29%. Detailed stall analysis indicates that memory dependencies are the leading cause of GPU idle time, accounting for up to 68% of stalls in certain kernels. This suggests that future optimizations should prioritize data alignment and locality over raw compute power.
Kernel Hotspots
Across all component benchmarks, six specific kernels—Convolution, GEMM, BatchNorm, ReLU, Element-wise, and Gradient—occupy over 80% of total running time. These "hotspots" provide a clear roadmap for systems engineers looking to optimize AI workloads.
Conclusion and Future Directions
AIBench provides a standardized lens through which to view AI in the wild, bridging the gap between standalone model accuracy and production-level system performance. While the current baseline uses general-purpose frameworks like Flask, which may have higher latency than proprietary industry solutions, it provides a vital open standard for research.
Future work involves expanding the suite to other verticals, such as clinical diagnosis in "Medicine AI," and refining data generators to better simulate the "data drift" encountered in live production environments.
Summary updated