Scaling Law for Recommendation Models: Towards General-purpose User Representations

Kyuyong Shin, Hanock Kwak, Kyung-Min Kim, Su Young Kim, Max Nihlén Ramström

TL;DR

This paper introduces CLUE, a Contrastive Learning User Encoder, that learns general-purpose user representations from massive, multi-service user behavior data. CLUE demonstrates strong transferability across diverse recommendation tasks and companies, even outperforming task-specific models, and reveals a power-law scaling behavior in user representation learning.

What Problem Does This Paper Address?

While large-scale pretrained models have revolutionized fields like vision and language, the area of general-purpose user representation learning for recommendation systems remains relatively underexplored. Existing approaches often focus on task-specific models or require fine-tuning, limiting their adaptability. The authors aim to bridge this gap by developing a universal user encoder that can learn transferable representations from diverse user behavior data, addressing questions about the scalability, transferability, and influencing factors of such models.

Methods

The core of CLUE is a hierarchical encoder consisting of an Item Transformer and a Service Transformer. The Item Transformer processes natural language descriptions of items in a user's behavior log, converting them into item embeddings. These item embeddings are then fed into the Service Transformer, which aggregates them to produce a user embedding for a specific service. CLUE employs a contrastive learning objective, treating representations of the same user across different services as positive pairs and representations of different users as negative pairs. This task-agnostic objective allows CLUE to learn generalizable representations. The authors also explore the impact of various scaling factors (data size, model capacity, sequence length, batch size) on performance and investigate the presence of a scaling law in user representation learning, where training error decreases as a power-law with computation.

Datasets

The paper utilizes a large-scale pretraining dataset comprising over 50 billion behavior tokens from 11 million users across search engine and e-commerce platforms. For downstream evaluation, they use:

  • Benchmark Datasets: "Books" and "Clothing Shoes and Jewelry" categories from the Amazon review dataset.
  • Industrial Datasets: Data from various services including e-commerce (PCR), web-based cartoons (FWR), news (NVR), marketing messages (MMR), and online travel agencies (OTAR).
  • Inter-Company-Level Transfer (ICLT) Dataset: Data from a beauty and cosmetic e-commerce platform from a different company, specifically designed to test cross-company transferability.

Key Insights and Ideas

  • Scaling Law in User Representation: The study empirically demonstrates that user representation learning exhibits a power-law scaling behavior with computation, similar to findings in NLP and vision. This suggests that increasing computational resources (model size, data, sequence length, batch size) can predictably improve performance.
  • Transferability Beyond Domains: CLUE's task-agnostic representations show remarkable transferability not only across different recommendation tasks but also across different companies, highlighting the potential for truly general-purpose user modeling.
  • Natural Language as a Unifier: Using natural language descriptions of items, rather than just IDs, is crucial for enabling flexible transfer across diverse services and companies.
  • Importance of Multiple Factors: Optimal performance requires scaling multiple factors (data size, model capacity, sequence length, batch size) in tandem, rather than focusing on just one.
  • Contrastive Learning for Generalization: The contrastive learning approach, applied across multiple services, effectively learns robust user representations that generalize well to unseen downstream tasks.

Key Results

  • CLUE significantly outperforms existing task-specific and pretraining-then-finetuning models on both benchmark and industrial downstream tasks, often by meaningful margins (e.g., over 6% improvement in MRR on MMR).
  • On an online PCR task, CLUE achieved a Click-Through-Rate (CTR) engagement increase of 7% compared to a baseline method, and comfortably outperformed a GNN model by 5%.
  • For the Inter-Company-Level Transfer task, CLUE showed a 2% higher MRR score than UserBERT, demonstrating its ability to generalize across company boundaries.
  • The paper shows that even with a reduced output dimension, CLUE maintains strong performance on downstream tasks, suggesting flexibility in model deployment.

Limitations and Future Work

  • Domain Constraints: The authors acknowledge that the domain of the pretraining data can still be a limitation for learning a truly complete general-purpose user representation.
  • Transferability Criteria: While they observe a correlation between domain similarity and performance, they haven't found a definitive criterion for predicting successful domain transfer. Future work could focus on developing such criteria.
  • "One-Size-Fits-All" Challenge: The paper suggests that a single "one-size-fits-all" approach to user modeling might still be challenging, and some downstream tasks may remain tricky.
  • Future Directions: The authors propose further research into the scaling laws for user representation learning, exploring more sophisticated methods for domain transfer, and investigating the broader impacts of CLUE in general recommender system applications.

Summary updated

Abstract

Recent advancement of large-scale pretrained models such as BERT, GPT-3, CLIP, and Gopher, has shown astonishing achievements across various task domains. Unlike vision recognition and language models, studies on general-purpose user representation at scale still remain underexplored. Here we explore the possibility of general-purpose user representation learning by training a universal user encoder at large scales. We demonstrate that the scaling law is present in user representation learning areas, where the training error scales as a power-law with the amount of computation. Our Contrastive Learning User Encoder (CLUE), optimizes task-agnostic objectives, and the resulting user embeddings stretch our expectation of what is possible to do in various downstream tasks. CLUE also shows great transferability to other domains and companies, as performances on an online experiment shows significant improvements in Click-Through-Rate (CTR). Furthermore, we also investigate how the model performance is influenced by the scale factors, such as training data size, model capacity, sequence length, and batch size. Finally, we discuss the broader impacts of CLUE in general.
BibTeX
@inproceedings{shin2023scaling,
  author = {Kyuyong Shin and Hanock Kwak and Kyung-Min Kim and Su Young Kim and Max Nihlén Ramström},
  title = {{Scaling Law for Recommendation Models: Towards General-purpose User Representations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i4.25582},
}
Figure