FeatureBox: Feature Engineering on GPUs for Massive-Scale Ads Systems

Weijie Zhao, Xuewu Jiao, Xinsheng Luo, Jingxue Li, Belhal Karimi, Ping Li

In the world of online advertising, a 0.1% increase in Click-Through Rate (CTR) accuracy can translate into millions of dollars in revenue. To chase these gains, engineers at companies like Baidu constantly experiment with new "features"—signals extracted from raw data logs (like user history or search queries). However, as models scale to trillions of parameters, the bottleneck has shifted from the training itself to the grueling process of preparing the data.

In their paper, FeatureBox, researchers from Baidu Research and Baidu Inc. present a novel framework that moves feature engineering directly onto GPU training servers, slashing end-to-end training times by up to 10x.

TL;DR

FeatureBox is an end-to-end training framework that pipelines feature extraction and model training on the same GPU servers. By eliminating the massive I/O overhead of traditional CPU-based data preparation, it achieves a 5-10x speedup for industrial-scale ads systems while saving hundreds of terabytes in intermediate storage.

What Problem Does This Paper Address?

In massive-scale ads systems, the standard workflow is fragmented. First, a massive CPU cluster (using MapReduce) processes raw logs (e.g., 50 TB) to extract features. This produces a mountain of intermediate data (up to 200 TB) that must be written to a distributed file system (HDFS) and then read back by GPU servers for training.

This "Stage-after-Stage" approach has three major flaws:

  1. I/O Bottleneck: Moving hundreds of terabytes between CPU clusters, HDFS, and GPU servers is incredibly slow.
  2. Resource Inefficiency: Maintaining separate CPU and GPU clusters for the same pipeline is expensive and complex.
  3. Iteration Slowdown: Because data preparation takes so long, researchers can't test new feature ideas quickly.

Methods

The authors propose FeatureBox, which integrates feature extraction into the training pipeline on GPU servers. Key technical innovations include:

  • Heterogeneous Operator Scheduling: Not all tasks belong on a GPU. FeatureBox uses a Directed Acyclic Graph (DAG) to sort operations. Computation-intensive tasks (like data cleaning and feature extraction) are rewritten as GPU operators, while memory-intensive tasks (like massive dictionary lookups that exceed GPU VRAM) are kept on the CPU.
  • Meta-Kernels: Launching thousands of tiny GPU tasks (kernels) creates significant overhead. FeatureBox groups operators into "meta-kernels" that execute sequentially within a single launch, minimizing "dead time" on the GPU.
  • Light-weight GPU Memory Management: Standard memory allocation (cudaMalloc) is too slow for the dynamic nature of feature extraction (e.g., processing strings of varying lengths). FeatureBox implements a block-level memory pool with a "single-pointer reset" strategy, allowing for near-instant memory allocation and reclamation.

Datasets

The framework was evaluated using two massive, real-world industrial datasets from Baidu’s advertising platform:

  • Application A: ~1 billion instances, 15 TB of raw logs.
  • Application B: ~2 billion instances, 25 TB of raw logs. These datasets represent the "massive-scale" reality of modern ads systems, involving trillions ($10^{12}$) of potential features.

Key Insights and Ideas

  • The I/O is the Enemy: The most significant performance gains didn't come from faster math, but from eliminating the middleman. By not writing intermediate features to HDFS, the system avoids the "I/O tax" entirely.
  • Pipelining is King: By processing data in mini-batches and feeding them directly into the training loop, the GPU is never "starved" for data.
  • Smart Hybridization: The authors didn't try to force everything onto the GPU. They recognized that the CPU’s access to massive system RAM is a feature, not a bug, for specific operations like large-scale table joins.

Key Results

FeatureBox was compared against a production-grade MapReduce + GPU Parameter Server baseline:

  • Massive Speedups: For Application B, the end-to-end time dropped from 27 hours to just 2.65 hours (a 10.19x speedup).
  • I/O Savings: The system saved approximately 50–100 TB of intermediate I/O per training run.
  • Efficiency: In a head-to-head feature extraction test, just 2 GPU servers outperformed a 30-node CPU cluster by more than 2x.
  • Scalability: The performance gap between FeatureBox and traditional methods grows as the data size increases, making it highly future-proof.

Limitations and Future Work

  • Memory Reset Constraints: The current memory management relies on a "reset-after-layer" approach. While efficient for the described DAGs, it might require more complex garbage collection for models with highly irregular or non-sequential data flows.
  • Hardware Specificity: The results are optimized for Baidu’s specific hierarchical GPU parameter server architecture. While the principles are general, porting this to a standard "All-Reduce" training setup (like NCCL) might require additional tuning.
  • Future Direction: The authors suggest exploring even more complex feature extraction logic, such as incorporating real-time graph-based features directly into the GPU pipeline.

Summary updated

Abstract

Deep learning has been widely deployed for online ads systems to predict click-through rate (CTR). Practitioners frequently re-train CTR models to test their new extracted features. As the CTR model training relies on a large number of raw input data logs, the feature extraction step takes a significant portion of the training time. In this paper, we propose FeatureBox, a novel end-to-end training framework that pipelines the feature extraction and the training on GPU servers to save the intermediate I/O of the feature extraction. We rewrite computation-intensive feature extraction operators as GPU operators and leave the memory-intensive operator on CPUs. We introduce a layer-wise operator scheduling algorithm to schedule these heterogeneous operators. We present a light-weight GPU memory management algorithm that supports dynamic GPU memory allocation with minimal overhead. We experimentally evaluate FeatureBox and compare it with the previous in-production feature extraction framework on two ads applications. The results confirm the effectiveness of our proposed method.
BibTeX
@inproceedings{zhao2022featurebox,
  author = {Weijie Zhao and Xuewu Jiao and Xinsheng Luo and Jingxue Li and Belhal Karimi and Ping Li},
  title = {{FeatureBox: Feature Engineering on GPUs for Massive-Scale Ads Systems}},
  booktitle = {IEEE Big Data},
  year = {2022},
  url = {https://openreview.net/forum?id=AkbwfJXeH5},
}

Loading related content...

Figure