PrefRec: Preference-based Recommender Systems for Reinforcing Long-term User Engagement

Wanqi Xue, Qingpeng Cai, Zhenghai Xue, Shuo Sun, Shuchang Liu, Dong Zheng, Peng Jiang, Bo AnarXiv 2022 · OpenReview

Abstract

Modern recommender systems often prioritize immediate metrics like Click-Through Rate (CTR), failing to foster long-term user "stickiness." PrefRec introduces a novel framework that replaces hand-crafted reward functions with a reward model learned from human preferences. By integrating Reinforcement Learning from Human Feedback (RLHF) with expectile regression, PrefRec optimizes for long-term engagement metrics such as visit frequency and session depth, significantly outperforming traditional reinforcement learning baselines.

Introduction: The Short-Sightedness of Modern Recommendations

Most commercial recommender systems are designed to maximize immediate engagement. While effective at driving clicks, these systems often struggle to cultivate long-term user loyalty. Reinforcement Learning (RL) is theoretically well-suited for long-term optimization; however, it faces a significant "reward engineering" challenge. Defining a precise mathematical formula that links a single recommendation to a user's return several days later is notoriously difficult. Existing solutions often rely on surrogate rewards (like clicks) that misalign with long-term goals or sparse rewards that lead to unstable learning.

The PrefRec Framework

PrefRec shifts the paradigm from manually defining rewards to learning them through user preferences. The core of the system is a preference-based RL (RLHF) pipeline that bypasses the need for complex scalar reward formulas.

The PrefRec framework overview, illustrating the transition from teacher-provided preferences over trajectories to a learned reward function and an Actor-Critic policy.

1. Learning from Preferences

Instead of assigning a scalar value to every action, PrefRec utilizes a "teacher" (which can be a human or a scripted proxy) to compare two sequences of user behavior, known as trajectories. For instance, if Trajectory A results in increased user activity while Trajectory B leads to churn, the teacher prefers A. These preferences are used to train a neural reward function based on the Bradley-Terry model, which assumes preference probability is proportional to the exponential of the sum of predicted rewards along those paths.

2. Optimizing for the Upper Tail

To ensure the system learns to exceed historical performance, PrefRec employs expectile regression (with a parameter $\tau > 0.5$) for training its Value-function (V-function). Standard regression typically targets the mean outcome, which can lead to a policy that merely imitates average historical data. By focusing on "above-average" outcomes, the agent learns to identify and replicate high-engagement behaviors found in the upper tail of the data distribution.

Visualization of expectile regression. By adjusting the parameter τ, the model can prioritize learning from high-performance outcomes (the upper tail) rather than just the mean.

Experimental Results

The authors evaluated PrefRec using a newly released dataset from Kuaishou, a leading short-form video platform. The dataset includes interaction histories for 100,000 users and over 25 million recommendation events. States are represented as 245-dimensional vectors covering user demographics and history, while actions are 8-dimensional item embeddings.

PrefRec was compared against a variety of baselines, including online RL (SAC, DDPG), offline RL (IQL, BCQ), and Imitation Learning. The results demonstrated PrefRec's clear superiority in fostering long-term engagement:

  • Session Depth: Achieved a 21% improvement over the strongest previous methods.
  • Visiting Frequency: Achieved a massive 55.2% improvement.

Ablation studies confirmed that both the expectile regression and the pre-training of the reward model are critical to these gains. Removing expectile regression (setting $\tau = 0.5$) resulted in a significant drop in performance, highlighting the importance of targeting the "upper tail" of the data distribution to move beyond simple data imitation.

Conclusion

PrefRec demonstrates that learning rewards from preferences is a viable and superior alternative to manual reward engineering for long-term engagement. By combining RLHF with expectile regression, the system provides a stable and effective way to optimize for complex, long-term user behaviors. While the current study primarily utilizes a scripted teacher in an offline setting, future work will focus on deploying PrefRec in live environments to handle the noise and subjectivity of real-time human preferences.

Summary updated

BibTeX
@misc{xue2022prefrec,
  author = {Wanqi Xue and Qingpeng Cai and Zhenghai Xue and Shuo Sun and Shuchang Liu and Dong Zheng and Peng Jiang and Bo An},
  title = {{PrefRec: Preference-based Recommender Systems for Reinforcing Long-term User Engagement}},
  year = {2022},
  url = {https://openreview.net/forum?id=ueLzUF_B4F},
}
Figure