Abstract
Deep learning has been widely deployed for online ads systems to predict Click-Through Rate (CTR). Machine learning researchers and practitioners frequently retrain CTR models to test their new extracted features. However, the CTR model training often relies on a large number of raw input data logs. Hence, the feature extraction can take a significant proportion of the training time for an industrial-level CTR model. In this paper, we propose FeatureBox, a novel end-to-end training framework that pipelines the feature extraction and the training on GPU servers to save the intermediate I/O of the feature extraction. We rewrite computation-intensive feature extraction operators as GPU operators and leave the memory-intensive operator on CPUs. We introduce a layer-wise operator scheduling algorithm to schedule these heterogeneous operators. We present a light-weight GPU memory management algorithm that supports dynamic GPU memory allocation with minimal overhead. We experimentally evaluate FeatureBox and compare it with the previous in-production feature extraction framework on two real-world ads applications. The results confirm the effectiveness of our proposed method.
Introduction
In the competitive landscape of online advertising, even a 0.1% improvement in Click-Through Rate (CTR) prediction accuracy can result in millions of dollars in incremental revenue. To achieve these gains, engineers at major platforms like Baidu continuously experiment with new features—signals derived from raw data logs such as user history, search queries, and context.
However, as models scale to trillions of parameters, the primary bottleneck has shifted. It is no longer just the training of the neural network itself, but the grueling process of preparing the data. Traditional workflows separate feature engineering and model training into distinct stages, often separated by massive I/O operations. FeatureBox addresses this by moving feature engineering directly onto GPU training servers, effectively placing data preparation on the "GPU express lane."
The Bottleneck: Stage-after-Stage Processing
In massive-scale ads systems, the standard workflow is fragmented. Typically, a massive CPU cluster using MapReduce processes raw logs (often exceeding 50 TB) to extract features. This stage produces a mountain of intermediate data—up to 200 TB—which must be written to a distributed file system (HDFS) and subsequently read back by GPU servers for training.
This "Stage-after-Stage" approach suffers from three critical flaws:
- I/O Bottleneck: Moving hundreds of terabytes between CPU clusters, HDFS, and GPU servers is incredibly time-consuming.
- Resource Inefficiency: Maintaining separate, massive CPU and GPU clusters for the same pipeline increases operational complexity and cost.
- Iteration Slowdown: The latency of data preparation prevents researchers from testing new feature ideas rapidly, stifling innovation.
The FeatureBox Framework
FeatureBox integrates feature extraction directly into the training pipeline on GPU servers. By eliminating the need to write intermediate features to external storage, the framework bypasses the "I/O tax" entirely. The system relies on three core technical innovations:
Heterogeneous Operator Scheduling
Not every task is suited for a GPU. FeatureBox utilizes a Directed Acyclic Graph (DAG) to organize operations. Computation-intensive tasks, such as data cleaning and complex feature transformations, are rewritten as GPU operators. Conversely, memory-intensive tasks—such as massive dictionary lookups that exceed GPU VRAM—are kept on the CPU, leveraging the system's high-capacity RAM.
Meta-Kernels
Launching thousands of small GPU kernels for individual feature operations creates significant scheduling overhead. FeatureBox mitigates this by grouping operators into "meta-kernels." These meta-kernels execute sequentially within a single launch, minimizing the "dead time" between GPU tasks and maximizing hardware utilization.
Light-weight GPU Memory Management
Standard memory allocation (e.g., cudaMalloc) is too slow for the dynamic nature of feature extraction, where string lengths and data shapes vary constantly. FeatureBox implements a block-level memory pool with a "single-pointer reset" strategy. This allows for near-instant memory allocation and reclamation, specifically optimized for the layer-wise flow of feature extraction.
Experimental Evaluation
The framework was evaluated using two massive, real-world industrial datasets from Baidu’s advertising platform:
- Application A: ~1 billion instances, 15 TB of raw logs.
- Application B: ~2 billion instances, 25 TB of raw logs.
These datasets represent the scale of modern ads systems, involving trillions ($10^{12}$) of potential features.
Key Results
FeatureBox was compared against a production-grade baseline consisting of a MapReduce cluster and a GPU Parameter Server. The results demonstrate significant improvements:
- Massive Speedups: For Application B, the end-to-end processing time dropped from 27 hours to just 2.65 hours—a 10.19x speedup.
- I/O Savings: The system eliminated approximately 50–100 TB of intermediate I/O per training run.
- Hardware Efficiency: In head-to-head feature extraction tests, just two GPU servers outperformed a 30-node CPU cluster by more than 2x.
- Scalability: The performance gap between FeatureBox and traditional methods widens as data size increases, suggesting the framework is well-suited for future growth in data volume.
Conclusion
FeatureBox demonstrates that the key to accelerating massive-scale machine learning lies in the tight integration of data preparation and model training. By treating feature engineering as a first-class citizen on the GPU and optimizing for heterogeneous execution, FeatureBox removes the I/O bottlenecks that have long hindered industrial CTR systems. This approach not only saves time and storage but also empowers researchers to iterate faster on the features that drive advertising revenue.
Summary updated