Merlin HugeCTR: GPU-accelerated Recommender System Training and Inference

Zehuan Wang, Yingcan Wei, Minseok Lee, Matthias Langer, Fan Yu, Jie Liu, Shijie Liu, Daniel G. Abel, Xu Guo, Jianbing Dong, Ji Shi, Kunlun Li

Abstract

Merlin HugeCTR is an open source, GPU-accelerated integration framework for click-through rate estimation. It optimizes both training and inference, whilst enabling model training at scale with model-parallel embeddings and data-parallel neural networks. In particular, Merlin HugeCTR combines a high-performance GPU embedding cache with an hierarchical storage architecture, to realize low-latency retrieval of embeddings for online model inference tasks. In the MLPerf v1.0 DLRM model training benchmark, Merlin HugeCTR achieves a speedup of up to 24.6x on a single DGX A100 (8x A100) over PyTorch on 4x4-socket CPU nodes. Merlin HugeCTR additionally features a hierarchical parameter server (HPS) and supports deployment via the NVIDIA Triton server framework, achieving a 5~62x speedup for popular recommendation models over CPU baseline implementations.

Introduction

Modern recommendation systems, particularly those used for Click-Through Rate (CTR) estimation, face a unique "memory wall." These models rely on massive embedding tables—look-up tables that translate categorical data (like User ID or Item ID) into numerical vectors. In industry-scale models, these tables can reach terabytes in size, far exceeding the memory capacity of individual GPUs. Standard deep learning frameworks often struggle with these sparse, memory-intensive operations at scale. Merlin HugeCTR addresses these challenges by providing a framework optimized for training across multiple GPUs and nodes and deploying for real-time inference with millisecond latency.

Architecture and Methods

HugeCTR introduces several architectural innovations to bridge the gap between massive data and GPU compute.

The Merlin HugeCTR software stack, highlighting the integration of training, inference, and the Hierarchical Parameter Server.

Hybrid Parallelism

The framework utilizes a hybrid approach: Model Parallelism (MP) for sparse embedding tables (distributing the table across GPUs) and Data Parallelism (DP) for dense neural network layers (replicating the model). This ensures efficient memory usage for large tables while maintaining high throughput for neural network computations.

Hierarchical Parameter Server (HPS) and Inference Workflow

For inference, HugeCTR employs a three-tier storage strategy to manage embeddings:

  1. Level 1 (GPU Cache): Stores "hot" (frequently accessed) embeddings.
  2. Level 2 (Volatile Database/VDB): Uses system RAM (e.g., Redis) for "warm" embeddings.
  3. Level 3 (Persistent Database/PDB): Uses SSDs for the entire "long-tail" of embeddings.
The end-to-end workflow from training to inference, showing real-time embedding updates via a message buffer and the hierarchical storage layers.

The system supports "Online Training," where incremental updates are pushed from the training cluster to the inference cluster in real-time via a message buffer (like Apache Kafka), ensuring the model remains current.

Key Results

The performance of Merlin HugeCTR was evaluated using the MLPerf v1.0 DLRM training benchmark and various popular model architectures like Wide & Deep (WDL) and DeepFM.

  • Training Performance: HugeCTR achieved a 24.6x speedup on a single NVIDIA DGX A100 compared to a 4x4-socket CPU node configuration.
  • Inference Throughput: The Hierarchical Parameter Server enabled a 5x to 62x speedup in inference throughput over CPU-only baselines, depending on the batch size.
  • Scalability: The framework successfully scales to multi-node environments, enabling the training of terabyte-scale models.

Conclusion and Future Work

Merlin HugeCTR significantly advances the state-of-the-art for large-scale recommendation systems. However, the optimal placement of embedding tables currently requires manual tuning based on data characteristics. Future work focuses on developing framework-agnostic embedding techniques, autonomous placement systems, and support for more flexible lookup types and upcoming GPU architectures.

Summary updated

Abstract

In this talk, we introduce Merlin HugeCTR. Merlin HugeCTR is an open source, GPU-accelerated integration framework for click-through rate estimation. It optimizes both training and inference, whilst enabling model training at scale with model-parallel embeddings and data-parallel neural networks. In particular, Merlin HugeCTR combines a high-performance GPU embedding cache with an hierarchical storage architecture, to realize low-latency retrieval of embeddings for online model inference tasks. In the MLPerf v1.0 DLRM model training benchmark, Merlin HugeCTR achieves a speedup of up to 24.6x on a single DGX A100 (8x A100) over PyTorch on 4x4-socket CPU nodes (4x4x28 cores). Merlin HugeCTR can also take advantage of multi-node environments to accelerate training even further. Since late 2021, Merlin HugeCTR additionally features a hierarchical parameter server (HPS) and supports deployment via the NVIDIA Triton server framework, to leverage the computational capabilities of GPUs for high-speed recommendation model inference. Using this HPS, Merlin HugeCTR users can achieve a 5~62x speedup (batch size dependent) for popular recommendation models over CPU baseline implementations, and dramatically reduce their end-to-end inference latency.
BibTeX
@misc{wang2022merlin,
  author = {Zehuan Wang and Yingcan Wei and Minseok Lee and Matthias Langer and Fan Yu and Jie Liu and Shijie Liu and Daniel G. Abel and Xu Guo and Jianbing Dong and Ji Shi and Kunlun Li},
  title = {{Merlin HugeCTR: GPU-accelerated Recommender System Training and Inference}},
  year = {2022},
  eprint = {2210.08803},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.08803},
}
Figure