The Architectural Implications of Facebook's DNN-Based Personalized Recommendation

Udit Gupta, Carole-Jean Wu, Xiaodong Wang, Maxim Naumov, Brandon Reagen, David Brooks, Bradford Cottel, Kim M. Hazelwood, Mark Hempstead, Bill Jia, Hsien-Hsin S. Lee, Andrey Malevich, Dheevatsa Mudigere, Mikhail Smelyanskiy, Liang Xiong, Xuan Zhang

Introduction

In recent years, the deep learning research community has focused heavily on optimizing Convolutional Neural Networks (CNNs) for vision and Recurrent Neural Networks (RNNs) for language. However, in large-scale production environments like Facebook’s data centers, the workload landscape is significantly different. This paper reveals that 79% of AI inference cycles are consumed by personalized recommendation models. These models present a unique "memory vs. compute" tug-of-war that challenges standard hardware optimization techniques and necessitates a fundamental rethink of data center architecture.

Figure 1: Cumulative distribution of AI inference cycles across production models at Facebook, showing the dominance of recommendation systems (RMCs).

The Nature of Recommendation Workloads

Unlike "dense" workloads such as image recognition, recommendation systems are inherently "sparse." They rely on embedding tables to map millions of categorical features—such as user IDs or content tags—into dense vectors. This architectural choice leads to massive model sizes, often consuming tens of gigabytes of memory, and irregular memory access patterns.

The paper characterizes three representative classes of production-scale recommendation models:

  • RMC1: A lightweight model used for initial "filtering" of thousands of items.
  • RMC2: A memory-intensive "ranking" model with many large embedding tables.
  • RMC3: A compute-intensive "ranking" model with large fully-connected (FC) layers.

As shown in the figure below, these recommendation models (RMCs) occupy a distinct region in the memory-compute landscape compared to traditional benchmarks like VGG16 or GNMT. They often require significant memory bandwidth relative to their total FLOPs, making them sensitive to memory latency and cache performance.

Figure 2: Analytical memory bytes read vs. analytical FLOPs for various AI models, highlighting the unique resource requirements of recommendation models (RMC1-3) compared to standard benchmarks.

Hardware Performance and Architectural Insights

The authors evaluated these models across three generations of Intel server CPUs: Haswell, Broadwell, and Skylake. Their findings challenge the assumption that newer hardware is always superior for all metrics.

Latency vs. Throughput

For single-request (unit batch) inference, the older Broadwell architecture often outperformed the newer Skylake. This is attributed to Broadwell’s higher base clock frequency. While Skylake introduces AVX-512 for wider SIMD support, these advantages are primarily realized in high-throughput, large-batch scenarios. In production environments where low latency is critical for user experience, the clock frequency remains a dominant factor.

Cache Hierarchy and Co-location

A critical architectural insight involves the shift from inclusive L3 caches (Haswell/Broadwell) to exclusive L3 caches (Skylake). Inclusive caches are more susceptible to performance degradation when multiple models are co-located on the same machine. This is due to higher L2 miss rates and back-invalidations, which can degrade latency by up to 2.6x for memory-intensive models like RMC2.

Key Results and Implications

The study provides several critical takeaways for the design of future AI infrastructure:

  1. Workload Dominance: Recommendation models are the primary drivers of AI infrastructure costs, accounting for nearly 80% of inference cycles.
  2. Latency Variance: There is a 15x variance in inference latency across different recommendation model classes on identical hardware, complicating scheduling and resource allocation.
  3. The Bottleneck Shift: Optimization efforts must look beyond matrix multiplication. While RMC3 is compute-heavy, RMC1 is bottlenecked by the SparseLengthsSum operator (embedding lookups). A "one-size-fits-all" accelerator is unlikely to be optimal.

Conclusion

This research underscores the need for "full-stack" hardware and software solutions that balance high-bandwidth memory requirements with high-throughput compute. To facilitate further research, the authors have open-sourced the DLRM (Deep Learning Recommendation Model) benchmark suite, providing the community with a tool to study production-scale recommendation workloads without requiring access to proprietary data. Future architectural designs must account for the irregular memory patterns and cache-interference sensitivities identified in this study to efficiently power the next generation of personalized services.

Summary updated

Abstract

The widespread application of deep learning has changed the landscape of computation in data centers. In particular, personalized recommendation for content ranking is now largely accomplished using deep neural networks. However, despite their importance and the amount of compute cycles they consume, relatively little research attention has been devoted to recommendation systems. To facilitate research and advance the understanding of these workloads, this paper presents a set of real-world, production-scale DNNs for personalized recommendation coupled with relevant performance metrics for evaluation. In addition to releasing a set of open-source workloads, we conduct in-depth analysis that underpins future system design and optimization for at-scale recommendation: Inference latency varies by 60% across three Intel server generations, batching and co-location of inference jobs can drastically improve latency-bounded throughput, and diversity across recommendation models leads to different optimization strategies.
BibTeX
@inproceedings{gupta2020architectural,
  author = {Udit Gupta and Carole-Jean Wu and Xiaodong Wang and Maxim Naumov and Brandon Reagen and David Brooks and Bradford Cottel and Kim M. Hazelwood and Mark Hempstead and Bill Jia and Hsien-Hsin S. Lee and Andrey Malevich and Dheevatsa Mudigere and Mikhail Smelyanskiy and Liang Xiong and Xuan Zhang},
  title = {{The Architectural Implications of Facebook's DNN-Based Personalized Recommendation}},
  booktitle = {HPCA},
  year = {2020},
  doi = {10.1109/hpca47549.2020.00047},
}
Figure