Abstract
Existing reinforcement learning (RL) recommenders often collapse complex user preferences into a single scalar reward, such as a star rating. This simplification loses critical nuance regarding why a user liked an item, leading to suboptimal long-term recommendations. We introduce Pareto Deterministic Policy Gradient (PDPG), a framework that treats distinct item aspects (e.g., price, location, or taste) as separate reward signals. By employing Pareto optimization, PDPG balances potentially conflicting objectives without introducing biased gradients. Our evaluation on three real-world datasets demonstrates that PDPG consistently outperforms state-of-the-art baselines in both accuracy and cumulative multi-aspect rewards.
Introduction
Most RL-based recommender systems model user feedback as a scalar reward. However, human preference is inherently multi-dimensional. For instance, two users might both provide a five-star rating for a hotel, but for entirely different reasons: User A may prioritize "room size," while User B values "low price."
When a system only observes the aggregate "5 stars," it fails to distinguish between these distinct user profiles. Furthermore, these aspects often conflict—improving "room size" typically necessitates an increase in "price." Standard RL models struggle to navigate these trade-offs, often defaulting to a simple average of scores that fails to capture "Pareto efficiency"—a state where one aspect cannot be improved without degrading another for a specific user.
The PDPG Framework
To address these challenges, we propose PDPG, which extends the Deterministic Policy Gradient (DPG) algorithm into a multi-objective setting through four key components:
- Multi-aspect Critics: Instead of a single "Critic" network estimating action value, the model employs $M$ critics, each dedicated to a specific item aspect (e.g., appearance, aroma, or palate).
- Pareto-efficient Actor: To update the "Actor" (the policy network), the model avoids simple averaging. Instead, it solves a Quadratic Programming (QP) problem to find a set of weights ensuring that the gradient update does not decrease the expected reward for any individual aspect.
- Supervised Regularization: To prevent the RL agent from exploring nonsensical item spaces, a supervised learning layer constrains the actor's output to remain close to items with which the user has historically interacted.
- Weight-reuse Mechanism: Theoretical analysis reveals that using mini-batches to calculate Pareto weights introduces a "gradient bias" proportional to $1/\sqrt{Z}$ (where $Z$ is batch size). PDPG implements a mechanism to store and reuse successful weights from previous steps, stabilizing training and lowering the error bound.
Experimental Evaluation
We evaluated PDPG on three real-world datasets containing explicit multi-aspect ratings: RateBeer (2.3M interactions, 5 aspects), BeerAdvocate (1.3M interactions, 5 aspects), and TripAdvisor (28k interactions, 7 aspects).
As shown in the results table, PDPG consistently outperformed baselines including Matrix Factorization (BPR), Deep Learning (NCF), and existing RL recommenders (DRR). Specifically, PDPG achieved a 7-9% improvement in NDCG@5 and F1@5 across all datasets.
Key Insights
- Balancing Conflicts: By embracing the contradictory nature of user preferences through Pareto optimization, PDPG avoids the "winner-take-all" problem where a model over-optimizes for a single dominant aspect at the expense of others.
- Long-term Engagement: Modeling aspects as long-term RL rewards allows the system to learn which specific features drive sustained user satisfaction rather than just immediate clicks.
- Theoretical Rigor: The introduction of the weight-reuse mechanism addresses the mathematical challenge of gradient bias in multi-objective RL, providing a robust foundation for training.
Conclusion
PDPG represents a significant step forward in making RL recommenders more sensitive to the nuances of human preference. By treating item aspects as independent yet related objectives, the framework achieves superior recommendation quality. Future work may explore applying this multi-aspect logic to system fairness, ensuring the interests of diverse stakeholders are balanced in a Pareto-efficient manner.
Summary updated