Reinforcement Recommendation with User Multi-aspect Preference

Xu Chen, Yali Du, Long Xia, Jun Wang

Abstract

Existing reinforcement learning (RL) recommenders often collapse complex user preferences into a single scalar reward, such as a star rating. This simplification loses critical nuance regarding why a user liked an item, leading to suboptimal long-term recommendations. We introduce Pareto Deterministic Policy Gradient (PDPG), a framework that treats distinct item aspects (e.g., price, location, or taste) as separate reward signals. By employing Pareto optimization, PDPG balances potentially conflicting objectives without introducing biased gradients. Our evaluation on three real-world datasets demonstrates that PDPG consistently outperforms state-of-the-art baselines in both accuracy and cumulative multi-aspect rewards.

Introduction

Most RL-based recommender systems model user feedback as a scalar reward. However, human preference is inherently multi-dimensional. For instance, two users might both provide a five-star rating for a hotel, but for entirely different reasons: User A may prioritize "room size," while User B values "low price."

The standard RL recommendation loop (a) and the challenge of multi-aspect preferences (b), where overall ratings mask distinct underlying priorities.

When a system only observes the aggregate "5 stars," it fails to distinguish between these distinct user profiles. Furthermore, these aspects often conflict—improving "room size" typically necessitates an increase in "price." Standard RL models struggle to navigate these trade-offs, often defaulting to a simple average of scores that fails to capture "Pareto efficiency"—a state where one aspect cannot be improved without degrading another for a specific user.

The PDPG Framework

To address these challenges, we propose PDPG, which extends the Deterministic Policy Gradient (DPG) algorithm into a multi-objective setting through four key components:

  1. Multi-aspect Critics: Instead of a single "Critic" network estimating action value, the model employs $M$ critics, each dedicated to a specific item aspect (e.g., appearance, aroma, or palate).
  2. Pareto-efficient Actor: To update the "Actor" (the policy network), the model avoids simple averaging. Instead, it solves a Quadratic Programming (QP) problem to find a set of weights ensuring that the gradient update does not decrease the expected reward for any individual aspect.
  3. Supervised Regularization: To prevent the RL agent from exploring nonsensical item spaces, a supervised learning layer constrains the actor's output to remain close to items with which the user has historically interacted.
  4. Weight-reuse Mechanism: Theoretical analysis reveals that using mini-batches to calculate Pareto weights introduces a "gradient bias" proportional to $1/\sqrt{Z}$ (where $Z$ is batch size). PDPG implements a mechanism to store and reuse successful weights from previous steps, stabilizing training and lowering the error bound.

Experimental Evaluation

We evaluated PDPG on three real-world datasets containing explicit multi-aspect ratings: RateBeer (2.3M interactions, 5 aspects), BeerAdvocate (1.3M interactions, 5 aspects), and TripAdvisor (28k interactions, 7 aspects).

Performance comparison of PDPG against baselines across three datasets, showing significant improvements in F1, NDCG, and cumulative rewards for individual aspects.

As shown in the results table, PDPG consistently outperformed baselines including Matrix Factorization (BPR), Deep Learning (NCF), and existing RL recommenders (DRR). Specifically, PDPG achieved a 7-9% improvement in NDCG@5 and F1@5 across all datasets.

Key Insights

  • Balancing Conflicts: By embracing the contradictory nature of user preferences through Pareto optimization, PDPG avoids the "winner-take-all" problem where a model over-optimizes for a single dominant aspect at the expense of others.
  • Long-term Engagement: Modeling aspects as long-term RL rewards allows the system to learn which specific features drive sustained user satisfaction rather than just immediate clicks.
  • Theoretical Rigor: The introduction of the weight-reuse mechanism addresses the mathematical challenge of gradient bias in multi-objective RL, providing a robust foundation for training.

Conclusion

PDPG represents a significant step forward in making RL recommenders more sensitive to the nuances of human preference. By treating item aspects as independent yet related objectives, the framework achieves superior recommendation quality. Future work may explore applying this multi-aspect logic to system fairness, ensuring the interests of diverse stakeholders are balanced in a Pareto-efficient manner.

Summary updated

Abstract

Formulating recommender system with reinforcement learning (RL) frameworks has attracted increasing attention from both academic and industry communities. While many promising results have been achieved, existing models mostly simulate the environment reward with a unified value, which may hinder the understanding of users’ complex preferences and limit the model performance. In this paper, we consider how to model user multi-aspect preferences in the context of RL-based recommender system. More specifically, we base our model on the framework of deterministic policy gradient (DPG), which is effective in dealing with large action spaces. A major challenge for modeling user multi-aspect preferences lies in the fact that they may contradict with each other. To solve this problem, we introduce Pareto optimization into the DPG framework. We assign each aspect with a tailored critic, and all the critics share the same actor. The Pareto optimization is realized by a gradient-based method, which can be easily integrated into the actor and critic learning process. Based on the designed model, we theoretically analyze its gradient bias in the optimization process, and we design a weight-reuse mechanism to lower the upper bound of this bias, which is shown to be effective for improving the model performance. We conduct extensive experiments based on three real-world datasets to demonstrate our model’s superiorities.
BibTeX
@inproceedings{chen2021reinforcement,
  author = {Xu Chen and Yali Du and Long Xia and Jun Wang},
  title = {{Reinforcement Recommendation with User Multi-aspect Preference}},
  booktitle = {The Web Conference},
  year = {2021},
  doi = {10.1145/3442381.3449846},
}
Figure