Developing a Recommendation Benchmark for MLPerf Training and Inference

Carole-Jean Wu, Robin Burke, Ed H. Chi, Joseph A. Konstan, Julian J. McAuley, Yves Raimond, Hao ZhangarXiv 2020 · arXiv

Abstract

Deep learning-based recommendation models (DLRMs) represent a significant portion of modern datacenter workloads, yet they remain under-represented in standardized performance benchmarks. This paper details the development of the MLPerf recommendation benchmark, addressing the "reality gap" between academic datasets and industrial requirements. We outline a framework that accounts for extreme sparsity, massive embedding tables, and the memory bottlenecks inherent in production-scale systems. By defining representative parameters and multi-scale benchmarks, this work aims to drive hardware and software optimization for the next generation of recommendation systems.

Introduction

While image classification and natural language processing have benefited from well-established benchmarks like ImageNet and GLUE, recommendation systems have historically lacked a standardized metric for performance evaluation. This is a critical oversight: recommendation tasks account for a massive portion of AI inference cycles in global datacenters. To address this, the MLPerf Recommendation Advisory Board—a coalition including experts from Facebook, Google, Netflix, and leading academic institutions—has developed a roadmap for a standardized, industry-relevant benchmark suite.

The "Reality Gap" in Recommendation Benchmarking

Existing academic benchmarks, such as MovieLens, often fail to reflect the challenges of production environments. The authors identify a "reality gap" characterized by several factors:

  1. Extreme Sparsity: Production systems must manage billions of items where only a few interactions occur per user. This creates a "needle in a haystack" problem that standard dense benchmarks do not capture.
  2. Power-Law Distributions: Interaction data follows a long-tail distribution. While some items are popular, the vast majority are infrequent but collectively consume significant system capacity.
  3. The Memory Wall: Industrial recommendation models utilize massive embedding tables that often exceed the memory capacity of standard hardware (16–256GB DRAM). Benchmarks must reflect the resulting pressure on memory bandwidth and capacity.

Methodology and Framework

The proposed MLPerf framework focuses on architectural diversity and representative system parameters to ensure the benchmark drives meaningful hardware and software innovation.

Architectural Diversity

The framework distinguishes between two primary model types:

  • Candidate Generation Models: These are latency-sensitive models that typically utilize dot-products or softmax operations to narrow down a large pool of items.
  • Ranking Models: These are interaction-heavy models that perform the final, fine-grained ordering of candidates.

Representative Parameters

To mirror industrial workloads, the board defined a "medium-scale" model as having 10–100 embedding tables, with 10K–10M rows per table and a latent space dimensionality ranging from 32 to 256. This ensures that the benchmark tests the system's ability to handle "dense-matrix multiplication with very sparse vectors," which is the hallmark of embedding lookups.

Dataset Evolution

A core component of the benchmark is the transition to more robust datasets. While the Criteo Kaggle dataset is more representative than MovieLens due to its higher sparsity, it is still considered small (approx. 1TB) compared to industrial scales. The authors advocate for the development of "at-scale" datasets spanning 10 to 100 billion rows to properly test distributed training systems and high-capacity memory architectures.

Key Specifications and Consensus

The MLPerf v0.7 specifications and beyond are built on industry consensus to ensure practical relevance:

  • Model Depth: Multi-Layer Perceptron (MLP) layers are generally restricted to a depth of less than 10 to reflect real-world latency constraints.
  • Feature Composition: A 50:50 split between dense and sparse features is recommended as a baseline for representative workloads.
  • Hashing: The board reached a consensus on a 50:50 split for hashing in embedding table access, mirroring common industry practices for managing large vocabularies.

Conclusion and Future Work

This benchmarking framework provides a foundation for evaluating the hardware and software stacks that power modern recommendation engines. However, the field is evolving rapidly. Future iterations of the MLPerf benchmark will need to incorporate:

  • Temporal Sequences: Moving beyond static snapshots to include RNNs, LSTMs, and Transformers (Attention layers) to capture evolving user interests.
  • Advanced Exploration: Incorporating Multi-Armed Bandits and Reinforcement Learning for dynamic item discovery.
  • Standardized Taxonomy: Developing a formal taxonomy to better categorize and compare future recommendation models.

By bridging the gap between academic research and industrial scale, this benchmark aims to catalyze the next generation of efficient, high-performance recommendation systems.

Summary updated

Abstract

Deep learning-based recommendation models are used pervasively and broadly, for example, to recommend movies, products, or other information most relevant to users, in order to enhance the user experience. Among various application domains which have received significant industry and academia research attention, such as image classification, object detection, language and speech translation, the performance of deep learning-based recommendation models is less well explored, even though recommendation tasks unarguably represent significant AI inference cycles at large-scale datacenter fleets. To advance the state of understanding and enable machine learning system development and optimization for the commerce domain, we aim to define an industry-relevant recommendation benchmark for the MLPerf Training andInference Suites. The paper synthesizes the desirable modeling strategies for personalized recommendation systems. We lay out desirable characteristics of recommendation model architectures and data sets. We then summarize the discussions and advice from the MLPerf Recommendation Advisory Board.
BibTeX
@misc{wu2020developing,
  author = {Carole-Jean Wu and Robin Burke and Ed H. Chi and Joseph A. Konstan and Julian J. McAuley and Yves Raimond and Hao Zhang},
  title = {{Developing a Recommendation Benchmark for MLPerf Training and Inference}},
  year = {2020},
  eprint = {2003.07336},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.07336},
}
Figure