Data Efficiency for Large Recommendation Models

Kshitij Jain, Jingru Xie, Kevin Regan, Cheng Chen, Jie Han, Steve Li, Zhuoshu Li, Todd Phillips, Myles Sussman, Matt Troup, Angel Yu, Jia ZhuoarXiv 2024 · DOI · arXiv

Introduction

In the landscape of online advertising, Large Recommendation Models (LRMs) serve as the critical engines for personalization and revenue. However, these models are notoriously data-hungry, often requiring training on hundreds of billions of examples. This "brute force" approach to scaling has led the industry to a point of diminishing returns, characterized by exorbitant computational costs and stifled research and development velocity.

Google researchers have addressed these challenges by shifting the focus from data quantity to data quality. By defining a "Chinchilla Law" for recommendation systems, they have established a roadmap for achieving model convergence with significantly less data, optimizing the balance between model parameters and training volume.

Methodology

The research introduces three core strategies to enhance data efficiency without compromising accuracy in live production environments:

  1. Continuous Distillation: Unlike traditional knowledge distillation, which typically occurs on static historical datasets, this method allows a "student" model to learn from a larger "teacher" model concurrently in a live environment. This ensures the student model captures the most recent patterns in user behavior.
  2. Continuous Downsampling: To manage label imbalance (e.g., unclicked vs. clicked ads), the authors propose a persistent downsampling strategy that continues into the online training stage. This uses uncertainty estimates to prioritize examples that provide the most informative signal to the model.
  3. Data Convergence & Pruning: By identifying a "convergence point"—the threshold where additional historical data ceases to improve accuracy—the researchers implemented periodic dataset pruning. This involves sliding the training window forward to focus exclusively on the most relevant, recent history.
Figure 3: Ranking loss comparison showing that continuous distillation maintains lower loss even as training data volume is reduced.

The RecSys "Chinchilla Law"

A pivotal contribution of this work is the application of iso-compute curves to recommendation models. Borrowing from Large Language Model (LLM) research, the authors sought to find the optimal sampling rate and model size for a fixed compute budget (e.g., 4,000 TPU hours).

Their findings reveal a clear "valley" in the loss curve, indicating a sweet spot for resource allocation. For their specific Ads CTR models, they identified an optimal sampling rate between 0.6 and 0.7. This demonstrates that simply training on more data is often less efficient than training on a carefully sampled subset with an appropriately sized model.

Figure 4: An Iso-Compute curve demonstrating the optimal training loss at a specific sampling rate for a fixed TPU budget.

Key Results and Impact

The implementation of these strategies yielded substantial efficiency gains in Google’s internal Ads CTR prediction models:

  • 35% Reduction in Data Volume: Continuous distillation allowed models to reach baseline accuracy levels while processing 35% less training data.
  • Superior Performance in Lean Scenarios: The performance gap between continuous distillation and traditional "cutoff" methods widened as data volume decreased, proving that these techniques are most valuable when computational resources are constrained.
  • Optimized Resource Allocation: The mapping of iso-compute curves provides a repeatable framework for engineers to pinpoint the most efficient training configurations, moving away from trial-and-error scaling.

Conclusion and Future Work

This research demonstrates that for LRMs, "more" is not always "better." By prioritizing data recency and implementing continuous processing techniques, developers can significantly reduce the environmental and financial costs of training while maintaining high performance.

Future work will involve exploring the overhead of maintaining concurrent teacher models and investigating whether these specific iso-compute curves generalize across other recommendation tasks, such as retrieval and diversity ranking. The shift toward "continuous" methodologies marks a significant step toward more sustainable and agile recommendation system development.

Summary updated

Abstract

Large recommendation models (LRMs) are fundamental to the multi-billion dollar online advertising industry, processing massive datasets of hundreds of billions of examples before transitioning to continuous online training to adapt to rapidly changing user behavior. The massive scale of data directly impacts both computational costs and the speed at which new methods can be evaluated (R&D velocity). This paper presents actionable principles and high-level frameworks to guide practitioners in optimizing training data requirements. These strategies have been successfully deployed in Google's largest Ads CTR prediction models and are broadly applicable beyond LRMs. We outline the concept of data convergence, describe methods to accelerate this convergence, and finally, detail how to optimally balance training data volume with model size.
BibTeX
@misc{jain2024data,
  author = {Kshitij Jain and Jingru Xie and Kevin Regan and Cheng Chen and Jie Han and Steve Li and Zhuoshu Li and Todd Phillips and Myles Sussman and Matt Troup and Angel Yu and Jia Zhuo},
  title = {{Data Efficiency for Large Recommendation Models}},
  year = {2024},
  eprint = {2410.18111},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.18111},
}
Figure