Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems

Weijie Zhao, Deping Xie, Ronglai Jia, Yulei Qian, Ruiquan Ding, Mingming Sun, Ping Li

In the world of online advertising, Click-Through Rate (CTR) prediction is the engine of revenue. Companies like Baidu, Google, and Bing process trillions of samples to train models that decide which ad you see. But there’s a massive engineering hurdle: these models are gargantuan. We’re talking about $10^{11}$ sparse features resulting in models that exceed 10 Terabytes in size—far too large for the memory of any single machine, or even the combined RAM of a small cluster.

In their MLSys paper, researchers from Baidu introduce a Distributed Hierarchical GPU Parameter Server that flips the script on traditional CPU-heavy training, proving that a 4-node GPU cluster can outperform a 150-node CPU cluster.

TL;DR

The authors present a 3-layer hierarchical parameter server (GPU HBM, CPU RAM, and SSD) designed to train 10TB+ deep learning models. By leveraging the sparsity of ad data and a 4-stage pipeline to hide I/O latency, their 4-node system achieves up to 4.8x the speed of a 150-node MPI cluster at a fraction of the cost.

What Problem Does This Paper Address?

Modern ad systems use "one-hot" or "multi-hot" binary features (e.g., user ID, query keywords). While the total feature space is massive ($10^{11}$), any single ad request only activates a tiny fraction of them. This creates a "memory wall":

  1. Size: The embedding layers (sparse parameters) are so large they don't fit in GPU memory or even the main CPU RAM of a typical node.
  2. Inefficiency: Traditional solutions use large MPI (Message Passing Interface) clusters with hundreds of CPU nodes. These are expensive to maintain, power-hungry, and suffer from massive communication overhead as nodes constantly swap parameters.
  3. Accuracy vs. Compression: While hashing can shrink models, the authors show that even a 0.1% drop in accuracy (AUC) leads to significant revenue loss, making "lossless" full-scale training a necessity.

Methods

The core innovation is a 3-layer hierarchical storage system that treats faster memory as a cache for slower, larger storage:

  1. HBM-PS (High-Bandwidth Memory): The GPU's onboard memory stores only the "working" parameters needed for the current training batch.
  2. MEM-PS (CPU RAM): Acts as a middle-tier cache, prefetching parameters from the SSD and buffering updates.
  3. SSD-PS (Solid State Drive): The "ground truth" storage where the full 10TB model lives.

Key Technical Feats:

  • 4-Stage Pipeline: To prevent the slow SSD from bottlenecking the fast GPU, the system overlaps four tasks: data transferring, parameter partitioning, SSD I/O, and GPU computation. While the GPU trains on batch $N$, the system is already pulling parameters for batch $N+1$ from the SSD.
  • Distributed GPU Hash Table: They implemented a custom hash table across multiple GPUs using NVLink and RDMA (Remote Direct Memory Access). This allows GPUs to pull parameters from each other’s memory without involving the CPU, eliminating a major traditional bottleneck.
  • SSD Management: Since SSDs hate random writes, the system writes updates in sequential batches and uses a background "compaction" thread (similar to LevelDB) to merge files and remove stale parameters.

Datasets

The system was evaluated using five real-world CTR prediction models from Baidu’s production environment (Models A through E).

  • Scale: Models ranged from 300 GB to 10 TB.
  • Features: Up to 500 billion non-zero weights.
  • Comparison: The baseline was Baidu's existing production MPI-cluster solution, which used between 75 and 150 CPU-only nodes depending on the model.

Key Insights and Ideas

  • The Sparsity Advantage: Even though the model is 10TB, a training batch only touches a few hundred megabytes of parameters. The hierarchy works because the "active" set is small enough to fit in GPU memory.
  • Hashing is a Double-Edged Sword: The authors experimented with a novel hashing method (OP+OSRP). While it reduced model size significantly, it consistently hurt AUC. In the ads business, where 0.1% AUC equals millions of dollars, the "brute force" approach of storing every parameter is actually the more profitable one.
  • SSD as "Slow RAM": By treating the SSD as the bottom of a memory hierarchy rather than just a filesystem, and using aggressive pipelining, they proved that disk latency can be almost entirely hidden during deep learning training.

Key Results

  • Speed: A 4-node GPU system (HPS-4) was 1.8x to 4.8x faster than the 150-node MPI cluster across all models.
  • Cost-Efficiency: When normalized for hardware and maintenance costs, the proposed system offers a 4x to 9x better price-performance ratio.
  • Accuracy: The training is "lossless." The AUC difference between the massive GPU system and the MPI baseline was within 0.1%, meaning no revenue was sacrificed for the speed gain.
  • Scalability: The system achieved a speedup of 3.57x when moving from 1 to 4 nodes, showing near-linear scaling.

Limitations and Future Work

  • Network Bottlenecks: While the system scales well, the authors noted that as the number of nodes increases, inter-node communication (RDMA) eventually becomes a limiting factor, leading to slightly sub-linear scaling.
  • SSD Wear: The paper doesn't deeply discuss the long-term endurance of SSDs under the constant write-heavy load of 10TB model updates, though their compaction strategy helps mitigate random write wear.
  • Integration: The system is being integrated into the PaddlePaddle ecosystem (as "PaddleBox"), suggesting future work will focus on making this hierarchical approach easier for the broader developer community to use.

Summary updated

Abstract

Neural networks of ads systems usually take input from multiple resources, e.g., query-ad relevance, ad features and user portraits. These inputs are encoded into one-hot or multi-hot binary features, with typically only a tiny fraction of nonzero feature values per example. Deep learning models in online advertising industries can have terabyte-scale parameters that do not fit in the GPU memory nor the CPU main memory on a computing node. For example, a sponsored online advertising system can contain more than $10^{11}$ sparse features, making the neural network a massive model with around 10 TB parameters. In this paper, we introduce a distributed GPU hierarchical parameter server for massive scale deep learning ads systems. We propose a hierarchical workflow that utilizes GPU High-Bandwidth Memory, CPU main memory and SSD as 3-layer hierarchical storage. All the neural network training computations are contained in GPUs. Extensive experiments on real-world data confirm the effectiveness and the scalability of the proposed system. A 4-node hierarchical GPU parameter server can train a model more than 2X faster than a 150-node in-memory distributed parameter server in an MPI cluster. In addition, the price-performance ratio of our proposed system is 4-9 times better than an MPI-cluster solution.
BibTeX
@inproceedings{zhao2020distributed,
  author = {Weijie Zhao and Deping Xie and Ronglai Jia and Yulei Qian and Ruiquan Ding and Mingming Sun and Ping Li},
  title = {{Distributed Hierarchical GPU Parameter Server for Massive Scale Deep Learning Ads Systems}},
  booktitle = {MLSys},
  year = {2020},
  url = {https://openreview.net/forum?id=AXOSezIjbjj},
}

Loading related content...

Figure