Reinforcing User Retention in a Billion Scale Short Video Recommender System

Qingpeng Cai, Shuchang Liu, Xueliang Wang, Tianyou Zuo, Wentao Xie, Bin Yang, Dong Zheng, Peng Jiang, Kun Gai

TL;DR

The paper introduces RLUR (Reinforcement Learning for User Retention), a framework that directly optimizes for long-term user retention in billion-scale short video systems like Kuaishou. By modeling user behavior as an infinite-horizon Markov Decision Process (MDP) and introducing novel techniques to handle the noise and long delays of retention signals, the authors achieved significant gains in Daily Active Users (DAU) and app open frequency.

What Problem Does This Paper Address?

Most recommender systems are optimized for immediate rewards, such as clicks, likes, or watch time. However, the ultimate goal for platforms like TikTok or Kuaishou is user retention—the probability that a user will return to the app.

Optimizing for retention is notoriously difficult for three reasons:

  1. Uncertainty: Retention is influenced by factors outside the app's control, like social events or a user's busy schedule.
  2. Bias: Highly active users naturally have higher retention, which can drown out the signal for less active users who actually need better recommendations to stay engaged.
  3. Long Delays: Unlike a "like" which happens instantly, the "reward" for retention (the user coming back) happens hours or days later. This creates a massive distribution shift between the policy being trained and the data being collected.

Methods

The authors formulate the problem as an infinite-horizon request-based MDP. In this setup, the "agent" (the recommender) chooses ranking weights for various scoring models at each user request to minimize the cumulative returning time (the time gap between sessions).

To solve the challenges mentioned above, they proposed the RLUR algorithm, which includes several key components:

  • Retention Critic Learning: They use a Deep Deterministic Policy Gradient (DDPG) approach to estimate the cumulative returning time.
  • Uncertainty Normalization: To reduce variance, they train a model to predict the probability of a user returning within a certain timeframe. The actual reward is then normalized against this prediction.
  • Bias Mitigation via User Grouping: They train separate policies for high-activity and low-activity user groups to ensure the model doesn't just cater to the "easy" active users.
  • Soft Regularization: To handle the long delay and resulting distribution shift, they introduce a soft regularization term. This term weights training samples based on how much the current policy deviates from the behavior policy that originally collected the data.
  • Heuristic & Intrinsic Rewards: Because retention is sparse, they use immediate feedback (like watch time) as a "heuristic" reward and Random Network Distillation (RND) as an "intrinsic" reward to encourage exploration of new states.
Figure 1: The infinite horizon request-based MDP and the framework of RLUR

Datasets

The researchers evaluated RLUR using two primary methods:

  1. Offline Simulator: Built using KuaiRand, a public, unbiased dataset of user interactions from Kuaishou. They developed a simulator to model immediate feedback, session exits, and return probabilities.
  2. Live Environment: A large-scale A/B test on the Kuaishou app, involving billions of users.

Key Insights and Ideas

  • Returning Time as the Metric: Instead of treating retention as a binary 0/1 (did they return?), they optimize for the time interval between visits. Minimizing this interval naturally increases app open frequency and DAU.
  • Dual-Critic Architecture: By maintaining separate critics for long-term retention ($Q_T$) and immediate engagement ($Q_I$), the model can balance short-term satisfaction with long-term loyalty.
  • Stabilizing the "Delay Gap": The soft regularization is a crucial engineering feat. It allows the model to learn from "stale" data (collected days ago) without the training becoming unstable due to policy drift.

Key Results

In live experiments against a strong Cross Entropy Method (CEM) baseline, RLUR showed consistent and significant improvements:

  • App Open Frequency: Increased by 0.450%.
  • Daily Active Users (DAU): Increased by 0.2%.
  • 1st-Day Retention: Improved by 0.053%.
  • 7th-Day Retention: Improved by 0.063%.

While these percentages seem small, the authors note that in a billion-scale system, a 0.1% increase in DAU is considered a major success. As shown in Figure 2, the performance gap between RLUR and the baseline continued to grow over a 100-day deployment period.

Live performance gap over 100 days, showing steady growth in open frequency, DAU, and retention.

Limitations and Future Work

The paper focuses heavily on the architectural and algorithmic side of RL for retention but leaves some areas for further exploration:

  • User State Complexity: While the state includes history and profiles, more complex representations of user "mood" or intent could potentially improve results.
  • Reward Shaping: The weights between the retention critic and the immediate reward critic ($\lambda_T, \lambda_I$) were set manually. Automated tuning of these hyperparameters could be a future direction.
  • Long-term vs. Short-term Trade-offs: The paper doesn't explicitly quantify the "cost" to immediate engagement (e.g., if watch time drops slightly to favor long-term retention), though the live results suggest both can improve simultaneously.

Summary updated

Abstract

Recently, short video platforms have achieved rapid user growth by recommending interesting content to users. The objective of the recommendation is to optimize user retention, thereby driving the growth of DAU (Daily Active Users). Retention is a long-term feedback after multiple interactions of users and the system, and it is hard to decompose retention reward to each item or a list of items. Thus traditional point-wise and list-wise models are not able to optimize retention. In this paper, we choose reinforcement learning methods to optimize the retention as they are designed to maximize the long-term performance. We formulate the problem as an infinite-horizon request-based Markov Decision Process, and our objective is to minimize the accumulated time interval of multiple sessions, which is equal to improving the app open frequency and user retention. However, current reinforcement learning algorithms can not be directly applied in this setting due to uncertainty, bias, and long delay time incurred by the properties of user retention. We propose a novel method, dubbed RLUR, to address the aforementioned challenges. Both offline and live experiments show that RLUR can significantly improve user retention. RLUR has been fully launched in Kuaishou app for a long time, and achieves consistent performance improvement on user retention and DAU.
BibTeX
@inproceedings{cai2023reinforcing,
  author = {Qingpeng Cai and Shuchang Liu and Xueliang Wang and Tianyou Zuo and Wentao Xie and Bin Yang and Dong Zheng and Peng Jiang and Kun Gai},
  title = {{Reinforcing User Retention in a Billion Scale Short Video Recommender System}},
  booktitle = {The Web Conference},
  year = {2023},
}
Figure