Exploring Scaling Laws of CTR Model for Online Performance Improvement

Weijiang Lai, Beihong Jin, Jiongyan Zhang, Yiyuan Zheng, Jian Dong, Jia Cheng, Jun Lei, Xingxing Wang

TL;DR

This paper introduces a new paradigm for improving Click-Through Rate (CTR) prediction models by leveraging the "scaling laws" observed in Large Language Models (LLMs). They propose a high-performance model, SUAN, that scales with model size and data, and then use knowledge distillation to create a lightweight, deployable version (LightSUAN) that retains superior performance and efficiency for online services.

What Problem Does This Paper Address?

Online personalized services heavily rely on Click-Through Rate (CTR) prediction models to estimate the likelihood of a user clicking on an item (like an ad or recommendation). While accuracy is paramount, these models also face stringent latency requirements for real-time deployment. Current CTR models often hit a performance bottleneck, where further improvements in accuracy come at the cost of significantly increased inference time, making them unsuitable for live systems. The authors aim to bridge this gap by exploring how scaling laws, successful in LLMs, can be applied to CTR models to achieve both high accuracy and efficient deployment.

Methods

The core of the paper lies in two main innovations:

  • SUAN (Stacked Unified Attention Network): This is a novel CTR model designed to exhibit scaling laws. Its key component is the Unified Attention Block (UAB), which unifies the modeling of sequential and non-sequential user behavior features. UAB employs multiple attention mechanisms, including self-attention for intra-sequence relationships, cross-attention with user profiles, and a dual alignment attention mechanism to adaptively fuse these representations. The network stacks multiple UABs to increase its "grade" (a combination of model size and sequence length).
  • LightSUAN and Online Distillation: To address the inference time issue of high-grade SUAN models, they create LightSUAN, a more efficient version. This is achieved through sparse self-attention (combining local and dilated attention to reduce computation) and parallel inference strategies. Crucially, they then employ online distillation to train LightSUAN. A high-grade SUAN acts as a "teacher" model, transferring its knowledge to the "student" LightSUAN, allowing the lightweight model to achieve superior performance without sacrificing speed.

Datasets

The authors used three datasets for their experiments:

  • Eleme: A 30-day behavior dataset from the Ele.me service.
  • Taobao: A 22-day behavior dataset from an Alibaba advertising system.
  • Industry: A large-scale industrial dataset with up to 1000 behavioral records per user over a year.

Key Insights and Ideas

  • Scaling Laws Apply to CTR: The paper demonstrates that CTR models, like LLMs, can exhibit performance improvements (measured by AUC) as model size, sequence length, and data size increase. This suggests a fundamental principle for building better CTR models.
  • Unified Attention for Richer Features: The UAB's ability to unify sequential and non-sequential features, and to weigh their importance from multiple perspectives (user profile, global context), is crucial for capturing complex user interests.
  • Distillation is Key for Deployment: High-performance models are often too slow for online use. Knowledge distillation provides a viable path to transfer the learned intelligence into a practical, low-latency model.
  • Sparse Attention and Parallel Inference are Essential: These techniques are not just optimizations; they are necessary enablers for deploying sophisticated models like SUAN in real-time environments.
  • Regularizers Matter for Scaling: Specific regularization techniques like RMSNorm and pre-norm strategies are identified as important for maintaining stable training and enabling consistent scaling behavior.

Key Results

  • SUAN Performance: SUAN significantly outperforms existing state-of-the-art CTR models across all three datasets, achieving the best AUC scores.
  • Scaling Law Validation: SUAN demonstrates scaling laws with respect to model grade (size and sequence length) and data size, with AUCs adhering to power-law functions.
  • Distilled LightSUAN Superiority: The distilled LightSUAN not only matches but often surpasses the performance of a SUAN model configured with one grade higher, while maintaining the efficiency of the original LightSUAN.
  • Online A/B Test Success: In a live online A/B test, the deployed distilled LightSUAN increased CTR by 2.81% and CPM by 1.69% compared to the baseline, with only a modest increase in average inference time.

Limitations and Future Work

  • Upper Bound on AUC: The paper suggests that there might be an upper bound for AUC improvement, even with increased model grade or data size, which is around 0.72.
  • Impact of Regularizers: While RMSNorm and pre-norm are shown to be beneficial for scaling, their precise theoretical underpinnings in the context of CTR scaling laws could be further explored.
  • Future Work: The authors implicitly suggest further exploration into optimizing the trade-off between model complexity, inference speed, and accuracy. The success of distillation also opens avenues for more advanced distillation techniques tailored for CTR prediction. The paper also highlights the importance of further research into the factors that enable and maintain scaling laws in CTR models.

Summary updated

Abstract

Click-Through Rate (CTR) models play a vital role in improving user experience and boosting business revenue in many online personalized services.However, current CTR models generally encounter bottlenecks in performance improvement.Inspired by the scaling law phenomenon of Large Language Models (LLMs), we propose a new paradigm for improving CTR predictions: first, constructing a CTR model with accuracy scalable to the model grade and data size, and then distilling the knowledge implied in this model into its lightweight model that can serve online users.To put it into practice, we construct a CTR model named SUAN (Stacked Unified Attention Network).In SUAN, we propose the unified attention block (UAB) as a behavior sequence encoder.A single UAB unifies the modeling of the sequential and non-sequential features and also measures the importance of each user behavior feature from multiple perspectives.Stacked UABs elevate the configuration to a high grade, paving the way for performance improvement.In order to benefit from the high performance of the high-grade SUAN and avoid the disadvantage of its long inference time, we modify the SUAN with sparse self-attention and parallel inference strategies to form LightSUAN, and then adopt online distillation to train the low-grade LightSUAN, taking a high-grade SUAN as a teacher.The distilled LightSUAN has superior performance but the same inference time as the LightSUAN, making it well-suited for online deployment.Experimental results show that SUAN performs exceptionally well and holds the scaling laws spanning three orders
BibTeX
@misc{lai2025exploring,
  author = {Weijiang Lai and Beihong Jin and Jiongyan Zhang and Yiyuan Zheng and Jian Dong and Jia Cheng and Jun Lei and Xingxing Wang},
  title = {{Exploring Scaling Laws of CTR Model for Online Performance Improvement}},
  year = {2025},
  eprint = {2508.15326},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.15326},
}
Figure