Abstract
Neural networks of ads systems usually take input from multiple resources, e.g., query-ad relevance, ad features and user portraits. These inputs are encoded into one-hot or multi-hot binary features, with typically only a tiny fraction of nonzero feature values per example. Deep learning models in online advertising industries can have terabyte-scale parameters that do not fit in the GPU memory nor the CPU main memory on a computing node. For example, a sponsored online advertising system can contain more than $10^{11}$ sparse features, making the neural network a massive model with around 10 TB parameters. In this paper, we introduce a distributed GPU hierarchical parameter server for massive scale deep learning ads systems. We propose a hierarchical workflow that utilizes GPU High-Bandwidth Memory, CPU main memory and SSD as 3-layer hierarchical storage. All the neural network training computations are contained in GPUs. Extensive experiments on real-world data confirm the effectiveness and the scalability of the proposed system. A 4-node hierarchical GPU parameter server can train a model more than 2X faster than a 150-node in-memory distributed parameter server in an MPI cluster. In addition, the price-performance ratio of our proposed system is 4-9 times better than an MPI-cluster solution.
Introduction
In the competitive landscape of online advertising, Click-Through Rate (CTR) prediction models are the primary drivers of revenue. These models often utilize over $10^{11}$ sparse features, resulting in parameter sets that exceed 10 Terabytes. This scale creates a significant "memory wall," as the models are far too large to fit into the High-Bandwidth Memory (HBM) of a single GPU or even the combined RAM of a standard compute node.
Traditional solutions rely on massive Message Passing Interface (MPI) clusters consisting of hundreds of CPU nodes. However, these clusters are expensive to maintain, power-hungry, and suffer from substantial communication overhead. While techniques like feature hashing can reduce model size, the authors demonstrate that even a 0.1% drop in AUC accuracy leads to significant revenue loss, making lossless, full-scale training a business necessity.
System Architecture
The core innovation of this work is a 3-layer hierarchical storage system that treats faster memory tiers as caches for larger, slower storage:
- HBM-PS (High-Bandwidth Memory): Stores only the active parameters required for the current training batch.
- MEM-PS (CPU RAM): Acts as a middle-tier cache, prefetching parameters from the SSD and buffering updates.
- SSD-PS (Solid State Drive): Serves as the primary storage for the complete 10TB model.
To prevent I/O bottlenecks, the system utilizes a 4-stage pipeline that overlaps data transfer, parameter partitioning, SSD I/O, and GPU computation. Furthermore, a distributed GPU hash table implemented with NVLink and RDMA allows GPUs to retrieve parameters from each other’s memory without CPU intervention, eliminating traditional latency points.
Experimental Evaluation
The system was evaluated using five real-world CTR prediction models from Baidu’s production environment, ranging from 300 GB to 10 TB in size. The proposed 4-node hierarchical GPU parameter server (HPS-4) was compared against a production-grade MPI cluster utilizing between 75 and 150 CPU nodes.
As shown in the results, HPS-4 achieved a training throughput 1.8x to 4.8x faster than the 150-node MPI cluster. When normalized for hardware and maintenance costs, the HPS system offers a 4x to 9x better price-performance ratio. Importantly, the training is lossless; the AUC accuracy remained within 0.1% of the baseline, ensuring no revenue sacrifice for the performance gains. The system also demonstrated near-linear scalability, achieving a 3.57x speedup when scaling from 1 to 4 nodes.
Conclusion
The Distributed Hierarchical GPU Parameter Server proves that by leveraging the inherent sparsity of ad data and a tiered memory architecture, a small GPU cluster can outperform massive CPU-based systems for terabyte-scale deep learning. By treating SSDs as the base layer of a memory hierarchy and employing aggressive pipelining, disk latency can be effectively hidden during training. Future work will focus on further optimizing inter-node RDMA communication and integrating the system into the broader PaddlePaddle ecosystem.
Summary updated