TL;DR
The paper introduces RLUR (Reinforcement Learning for User Retention), a framework that directly optimizes for long-term user retention in billion-scale short video systems like Kuaishou. By modeling user behavior as an infinite-horizon Markov Decision Process (MDP) and introducing novel techniques to handle the noise and long delays of retention signals, the authors achieved significant gains in Daily Active Users (DAU) and app open frequency.
What Problem Does This Paper Address?
Most recommender systems are optimized for immediate rewards, such as clicks, likes, or watch time. However, the ultimate goal for platforms like TikTok or Kuaishou is user retention—the probability that a user will return to the app.
Optimizing for retention is notoriously difficult for three reasons:
- Uncertainty: Retention is influenced by factors outside the app's control, like social events or a user's busy schedule.
- Bias: Highly active users naturally have higher retention, which can drown out the signal for less active users who actually need better recommendations to stay engaged.
- Long Delays: Unlike a "like" which happens instantly, the "reward" for retention (the user coming back) happens hours or days later. This creates a massive distribution shift between the policy being trained and the data being collected.
Methods
The authors formulate the problem as an infinite-horizon request-based MDP. In this setup, the "agent" (the recommender) chooses ranking weights for various scoring models at each user request to minimize the cumulative returning time (the time gap between sessions).
To solve the challenges mentioned above, they proposed the RLUR algorithm, which includes several key components:
- Retention Critic Learning: They use a Deep Deterministic Policy Gradient (DDPG) approach to estimate the cumulative returning time.
- Uncertainty Normalization: To reduce variance, they train a model to predict the probability of a user returning within a certain timeframe. The actual reward is then normalized against this prediction.
- Bias Mitigation via User Grouping: They train separate policies for high-activity and low-activity user groups to ensure the model doesn't just cater to the "easy" active users.
- Soft Regularization: To handle the long delay and resulting distribution shift, they introduce a soft regularization term. This term weights training samples based on how much the current policy deviates from the behavior policy that originally collected the data.
- Heuristic & Intrinsic Rewards: Because retention is sparse, they use immediate feedback (like watch time) as a "heuristic" reward and Random Network Distillation (RND) as an "intrinsic" reward to encourage exploration of new states.
Datasets
The researchers evaluated RLUR using two primary methods:
- Offline Simulator: Built using KuaiRand, a public, unbiased dataset of user interactions from Kuaishou. They developed a simulator to model immediate feedback, session exits, and return probabilities.
- Live Environment: A large-scale A/B test on the Kuaishou app, involving billions of users.
Key Insights and Ideas
- Returning Time as the Metric: Instead of treating retention as a binary 0/1 (did they return?), they optimize for the time interval between visits. Minimizing this interval naturally increases app open frequency and DAU.
- Dual-Critic Architecture: By maintaining separate critics for long-term retention ($Q_T$) and immediate engagement ($Q_I$), the model can balance short-term satisfaction with long-term loyalty.
- Stabilizing the "Delay Gap": The soft regularization is a crucial engineering feat. It allows the model to learn from "stale" data (collected days ago) without the training becoming unstable due to policy drift.
Key Results
In live experiments against a strong Cross Entropy Method (CEM) baseline, RLUR showed consistent and significant improvements:
- App Open Frequency: Increased by 0.450%.
- Daily Active Users (DAU): Increased by 0.2%.
- 1st-Day Retention: Improved by 0.053%.
- 7th-Day Retention: Improved by 0.063%.
While these percentages seem small, the authors note that in a billion-scale system, a 0.1% increase in DAU is considered a major success. As shown in Figure 2, the performance gap between RLUR and the baseline continued to grow over a 100-day deployment period.
Limitations and Future Work
The paper focuses heavily on the architectural and algorithmic side of RL for retention but leaves some areas for further exploration:
- User State Complexity: While the state includes history and profiles, more complex representations of user "mood" or intent could potentially improve results.
- Reward Shaping: The weights between the retention critic and the immediate reward critic ($\lambda_T, \lambda_I$) were set manually. Automated tuning of these hyperparameters could be a future direction.
- Long-term vs. Short-term Trade-offs: The paper doesn't explicitly quantify the "cost" to immediate engagement (e.g., if watch time drops slightly to favor long-term retention), though the live results suggest both can improve simultaneously.
Summary updated