Cold & Warm Net: Addressing Cold-Start Users in Recommender Systems

Xiangyu Zhang, Zongqiang Kuang, Zehao Zhang, Fan Huang, Xianfeng Tan

In the world of industrial recommender systems, the "cold-start" problem is a perennial thorn in the side of engineers. How do you recommend relevant content to a user who has just joined the platform and has zero or very few interactions? While many solutions exist for the ranking stage, the matching (retrieval) stage—where we must quickly narrow down millions of items to a few thousand—presents a unique challenge of scale and speed.

A team from Tencent and Tsinghua University recently published Cold & Warm Net, a novel architecture designed specifically to handle user cold-starts during this critical matching phase.

TL;DR

Cold & Warm Net introduces a dual-expert architecture that dynamically balances between "cold-start" and "warm-up" models using a gate network. By employing Dynamic Knowledge Distillation and an explicit Bias Net, it effectively transfers knowledge from active users to newcomers, significantly improving retrieval accuracy and user retention in real-world production environments.

What Problem Does This Paper Address?

Most industrial recommender systems operate in two stages: Matching (retrieval) and Ranking. While the ranking stage can afford complex models to handle cold-start users, the matching stage must be incredibly efficient to handle billion-scale item pools.

Existing methods like meta-learning are often too computationally expensive for the matching stage. Meanwhile, standard "two-tower" models (like DSSM or YouTubeDNN) struggle with cold-start users because they rely heavily on interaction history to learn user embeddings. If a user has no history, their embedding is essentially noise, leading to poor recommendations and, ultimately, user churn.

Methods

The authors propose a "Cold & Warm Net" consisting of several innovative components:

  1. Dual-Expert Strategy: Instead of one model for everyone, they use two:
  2. Cold-start Expert: Focuses on user profile features (age, gender) and "user group" embeddings (prior knowledge from similar active users).
  3. Warm-up Expert: Focuses on user action features (recent clicks, watch history).
  4. Gate Network: A lightweight network that looks at the user's state (e.g., login frequency, activity level) and decides how much to trust each expert. This allows for a smooth, dynamic transition as a user moves from "cold" to "warm."
  5. Dynamic Knowledge Distillation (DKD): This is the "teacher-student" part. To help the cold-start expert learn better representations, it "distills" knowledge from the warm-up expert. Crucially, it uses a Teacher Selector: it only distills information if the warm-up expert is actually performing better on a specific sample, preventing the cold-start model from learning "bad" or irrelevant habits.
  6. Bias Net: Cold-start users naturally have lower click rates than power users. The Bias Net explicitly models this behavioral discrepancy using features selected via Mutual Information, ensuring the main model focuses on interest rather than just activity levels.

Datasets

The model was evaluated on two distinct datasets:

  • MovieLens 1M: The classic public benchmark for recommendation algorithms.
  • Little-World: A massive, real-world dataset collected from a Tencent short video platform, containing over 433,000 users and 15 million interactions. This dataset provides a more realistic look at industrial-scale challenges.

Key Insights and Ideas

  • The Spectrum of "Coldness": Users aren't just binary (Cold vs. Warm). The use of a Gate Network acknowledges that user states are fluid and change as they interact with the app.
  • Selective Distillation: Standard knowledge distillation can sometimes lead to "assimilation," where the student just mimics the teacher's mistakes. The Teacher Selector in DKD is an "aha moment"—it ensures the cold-start expert only learns from the warm-up expert when the teacher actually knows the right answer.
  • Separating Interest from Bias: By using a dedicated Bias Net, the authors decouple the probability of an action (which is lower for new users) from the relevance of the content.

Key Results

The Cold & Warm Net was tested against industry standards like DSSM, YouTubeDNN, and state-of-the-art multi-interest models (Mind, UMI).

  • Offline Performance: On the Little-World dataset, Cold & Warm Net improved the Hit Rate (HR@100) for cold-start users by 23.94% and NDCG@10 by 35.22% compared to the strongest baseline.
  • Online Impact: After deployment on a major short video platform, the model led to a 3.27% increase in App Dwell Time (APT) and a 1.01% increase in User Retention Rate (URR). In the world of high-traffic apps, these are massive gains.
  • Ablation Studies: The authors proved that both DKD and the Bias Net were essential; removing either led to a significant drop in performance for cold-start users.

Limitations and Future Work

While the paper demonstrates impressive results, a few areas remain for exploration:

  • Scalability Overhead: While designed for the matching stage, the dual-expert system and gate network add more parameters than a standard single-tower model. The authors claim it meets scalability requirements, but the specific latency trade-offs aren't detailed.
  • Teacher Complexity: The current DKD uses the warm-up expert as the teacher. Future work could explore using an even more powerful "cross-domain" teacher or a ranking-stage model to guide the matching-stage student.
  • Data Availability: The "Little-World" dataset is currently internal to Tencent, though the authors mention a potential future public release.

For researchers and engineers working on retrieval, Cold & Warm Net provides a robust blueprint for moving beyond "one-size-fits-all" embeddings and treating user lifecycle stages as a dynamic learning problem.

Summary updated

Abstract

Cold-start recommendation is one of the major challenges faced by recommender systems (RS). Herein, we focus on the user cold-start problem. Recently, methods utilizing side information or meta-learning have been used to model cold-start users. However, it is difficult to deploy these methods to industrial RS. There has not been much research that pays attention to the user cold-start problem in the matching stage. In this paper, we propose Cold & Warm Net based on expert models who are responsible for modeling cold-start and warm-up users respectively. A gate network is applied to incorporate the results from two experts. Furthermore, dynamic knowledge distillation acting as a teacher selector is introduced to assist experts in better learning user representation. With comprehensive mutual information, features highly relevant to user behavior are selected for the bias net which explicitly models user behavior bias. Finally, we evaluate our Cold & Warm Net on public datasets in comparison to models commonly applied in the matching stage and it outperforms other models on all user types. The proposed model has also been deployed on an industrial short video platform and achieves a significant increase in app dwell time and user retention rate.
BibTeX
@misc{zhang2023cold,
  author = {Xiangyu Zhang and Zongqiang Kuang and Zehao Zhang and Fan Huang and Xianfeng Tan},
  title = {{Cold \& Warm Net: Addressing Cold-Start Users in Recommender Systems}},
  year = {2023},
  eprint = {2309.15646},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.15646},
}
Figure