On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models

Rohan Anil, Sandra Gadanho, Da Huang, Nijith Jacob, Zhuoshu Li, Dong Lin, Todd Phillips, Cristina Pop, Kevin Regan, Gil I. Shamir, Rakesh Shivanna, Qiqi Yan

This paper offers a rare "tour of the factory floor" for one of the world’s largest machine learning systems: Google’s search ads click-through rate (CTR) prediction model. Rather than focusing on a single novel algorithm, the authors detail a suite of practical engineering techniques required to maintain a system that handles billions of weights, trains on over 100 billion examples, and serves more than 100,000 requests per second.

TL;DR

Google engineers detail the "switch-back" cycle of industrial ML: deploying high-impact accuracy techniques (like second-order optimization and rank losses) and then neutralizing their massive computational costs through efficiency methods (like AutoML and data sampling). The paper highlights how production-specific challenges like model irreproducibility and UI-driven bias are solved through smooth activation functions and model factorization.

What Problem Does This Paper Address?

In the world of search advertising, predicting CTR is the engine of the business. However, moving from a research paper to a production system at this scale introduces three massive hurdles:

  1. The Efficiency-Accuracy Trade-off: A 1% accuracy gain is worthless if it doubles serving latency or training costs.
  2. Non-Stationarity: User behavior and ad relevance change by the minute, requiring a single-pass, online learning approach rather than static batch training.
  3. System Irreproducibility: Deep neural networks are non-convex. Training the same model twice on the same data can lead to different predictions. In an auction system, this "churn" causes unstable pricing and unpredictable user experiences.

Methods

The authors describe a "switch-back" philosophy (illustrated in Figure 1) where they alternatingly invest in quality and efficiency.

Figure 1: "Switch-backs" of incremental costly quality-improving techniques and efficiency methods. (Illustration not to any scale.)

Key Techniques:

  • Distributed Shampoo: A second-order optimizer that uses second derivatives for faster convergence. To make this computationally feasible, they pipeline the heavy math (inverse-pth roots) to CPUs while the TPUs handle the gradients (Figure 3).
  • Model Factorization: To handle the "UI effect" (where an ad's position affects its CTR), they split the model into a heavy Quality DNN (evaluated once per ad) and a lightweight UI DNN (evaluated hundreds of times for different placements). This is shown in Figure 4.
  • SmeLU (Smooth ReLU): They replaced standard ReLU activations with a smooth version to reduce the "ruggedness" of the loss landscape, significantly improving model reproducibility.
  • Bias Constraints: They add Lagrangian dual variables to the objective function to anchor the model to a calibrated solution, ensuring that the predicted CTR matches reality across different slices of data.
  • AutoML for Efficiency: They use weight-sharing Neural Architecture Search (NAS) to automatically find the optimal widths for layers and embeddings that stay within a strict "machine cost" budget (Figure 2).
Figure 2: Weight-sharing based NAS with cost constraints.

Datasets

The paper uses Google's internal search ads logs. The scale is immense:

  • Training Data: Over 100 billion examples.
  • Evaluation: They use Progressive Validation, where a model predicts on an example before it is allowed to train on it. This mimics the real-world deployment where models must generalize to the immediate future.

Key Insights and Ideas

  • Second-Order Optimization is Production-Ready: The successful deployment of Distributed Shampoo is a milestone. By offloading preconditioner math to host CPUs, they achieved a 0.44% accuracy boost—massive in this domain—with only a 10% increase in training time.
  • Reproducibility is a First-Class Metric: The authors use "Relative Prediction Difference" (PD) to measure how much two identical models diverge. They found that standard tricks like fixed initialization don't work at scale, but smoothing the activation function (SmeLU) does.
  • Calibration > Accuracy: In an auction, it's not enough to know which ad is better; you must know how much better. Bias constraints allow them to force the model to be calibrated even on "out-of-distribution" slices where data is sparse.

Key Results

The authors provide a breakdown of how different techniques impacted the production model (summarized in Table 1).

Table 1: Summary of accuracy and cost impacts.
  • Rank Losses: Provided the largest accuracy gain (+0.81%) with negligible cost.
  • Data Sampling: Reduced the training dataset to <25% of its original size without significant loss in accuracy.
  • AutoML: Reduced training time per step by 16% while maintaining neutral accuracy.
  • Bias Constraints: Reduced "Ads/Query Churn" (a measure of system instability) by 85%.

Limitations and Future Work

The authors acknowledge that while these techniques work for Google's specific regime (sparse features, online learning, TPU-heavy infrastructure), they may not translate directly to other domains like computer vision. A major ongoing challenge is credit attribution: disentangling whether a click happened because the ad was good or simply because it was placed in a prominent position. While model factorization helps, perfectly disentangling these correlations remains an open research problem.

Summary updated

Abstract

For industrial-scale advertising systems, prediction of ad click-through rate (CTR) is a central problem. Ad clicks constitute a significant class of user engagements and are often used as the primary signal for the usefulness of ads to users. Additionally, in cost-per-click advertising systems where advertisers are charged per click, click rate expectations feed directly into value estimation. Accordingly, CTR model development is a significant investment for most Internet advertising companies. Engineering for such problems requires many machine learning (ML) techniques suited to online learning that go well beyond traditional accuracy improvements, especially concerning efficiency, reproducibility, calibration, credit attribution. We present a case study of practical techniques deployed in Google's search ads CTR model. This paper provides an industry case study highlighting important areas of current ML research and illustrating how impactful new ML methods are evaluated and made useful in a large-scale industrial setting.
BibTeX
@inproceedings{anil2022factory,
  author = {Rohan Anil and Sandra Gadanho and Da Huang and Nijith Jacob and Zhuoshu Li and Dong Lin and Todd Phillips and Cristina Pop and Kevin Regan and Gil I. Shamir and Rakesh Shivanna and Qiqi Yan},
  title = {{On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models}},
  booktitle = {ORSUM@RecSys},
  year = {2022},
}
Figure