Abstract
Merlin HugeCTR is an open source, GPU-accelerated integration framework for click-through rate estimation. It optimizes both training and inference, whilst enabling model training at scale with model-parallel embeddings and data-parallel neural networks. In particular, Merlin HugeCTR combines a high-performance GPU embedding cache with an hierarchical storage architecture, to realize low-latency retrieval of embeddings for online model inference tasks. In the MLPerf v1.0 DLRM model training benchmark, Merlin HugeCTR achieves a speedup of up to 24.6x on a single DGX A100 (8x A100) over PyTorch on 4x4-socket CPU nodes. Merlin HugeCTR additionally features a hierarchical parameter server (HPS) and supports deployment via the NVIDIA Triton server framework, achieving a 5~62x speedup for popular recommendation models over CPU baseline implementations.
Introduction
Modern recommendation systems, particularly those used for Click-Through Rate (CTR) estimation, face a unique "memory wall." These models rely on massive embedding tables—look-up tables that translate categorical data (like User ID or Item ID) into numerical vectors. In industry-scale models, these tables can reach terabytes in size, far exceeding the memory capacity of individual GPUs. Standard deep learning frameworks often struggle with these sparse, memory-intensive operations at scale. Merlin HugeCTR addresses these challenges by providing a framework optimized for training across multiple GPUs and nodes and deploying for real-time inference with millisecond latency.
Architecture and Methods
HugeCTR introduces several architectural innovations to bridge the gap between massive data and GPU compute.
Hybrid Parallelism
The framework utilizes a hybrid approach: Model Parallelism (MP) for sparse embedding tables (distributing the table across GPUs) and Data Parallelism (DP) for dense neural network layers (replicating the model). This ensures efficient memory usage for large tables while maintaining high throughput for neural network computations.
Hierarchical Parameter Server (HPS) and Inference Workflow
For inference, HugeCTR employs a three-tier storage strategy to manage embeddings:
- Level 1 (GPU Cache): Stores "hot" (frequently accessed) embeddings.
- Level 2 (Volatile Database/VDB): Uses system RAM (e.g., Redis) for "warm" embeddings.
- Level 3 (Persistent Database/PDB): Uses SSDs for the entire "long-tail" of embeddings.
The system supports "Online Training," where incremental updates are pushed from the training cluster to the inference cluster in real-time via a message buffer (like Apache Kafka), ensuring the model remains current.
Key Results
The performance of Merlin HugeCTR was evaluated using the MLPerf v1.0 DLRM training benchmark and various popular model architectures like Wide & Deep (WDL) and DeepFM.
- Training Performance: HugeCTR achieved a 24.6x speedup on a single NVIDIA DGX A100 compared to a 4x4-socket CPU node configuration.
- Inference Throughput: The Hierarchical Parameter Server enabled a 5x to 62x speedup in inference throughput over CPU-only baselines, depending on the batch size.
- Scalability: The framework successfully scales to multi-node environments, enabling the training of terabyte-scale models.
Conclusion and Future Work
Merlin HugeCTR significantly advances the state-of-the-art for large-scale recommendation systems. However, the optimal placement of embedding tables currently requires manual tuning based on data characteristics. Future work focuses on developing framework-agnostic embedding techniques, autonomous placement systems, and support for more flexible lookup types and upcoming GPU architectures.
Summary updated