TL;DR
This paper introduces DeepPage, a novel framework that uses deep reinforcement learning to optimize entire pages of recommended items, not just individual items. By jointly considering item selection and their 2D display, DeepPage significantly improves recommendation quality by better capturing user preferences and long-term engagement.
What Problem Does This Paper Address?
Traditional recommender systems often struggle with the dynamic nature of user preferences and the complexity of presenting recommendations. In e-commerce, users typically see a "page" of items, and their feedback (skipping, clicking, purchasing) informs the next set of recommendations. This process presents two key challenges:
- Updating Recommendation Strategy in Real-Time: User preferences can change rapidly. Static recommendation strategies fail to adapt, leading to suboptimal suggestions.
- Generating Pages with Proper Display: Recommendations aren't just a list of items; their arrangement on a 2D page matters. Users browse in chunks, and the spatial layout can influence their engagement. Existing methods often treat this as a 1D ranking problem and generate item sets and display strategies separately, leading to sub-optimal combinations.
This paper aims to address these challenges by developing a system that can simultaneously generate a set of complementary items and an optimal 2D display strategy, adapting to user feedback in real-time.
Methods
The core of the proposed solution is the DeepPage framework, built upon an Actor-Critic deep reinforcement learning architecture.
- Actor-Critic Framework: This architecture is chosen for its ability to handle large and dynamic action spaces (the vast number of possible item pages) and to learn optimal strategies by maximizing long-term rewards, rather than just immediate ones.
- The Actor component is responsible for generating the recommendation page (the action). It uses an Encoder-Decoder architecture.
- Encoder: Captures user preferences. It uses a GRU (Gated Recurrent Unit) to learn an initial user preference from their history and another GRU with an attention mechanism to capture real-time preferences based on current interactions. Crucially, it incorporates item embeddings, category information, and user feedback.
- Decoder: Generates the recommendation page. It uses a deconvolutional neural network (DeCNN) to reconstruct a 2D page of item recommendations from the learned user preference representation.
- The Critic component evaluates the action taken by the Actor. It learns an action-value function (Q-value) that estimates the quality of a recommended page given the user's current state. This judgment is then used to update the Actor's parameters.
- Page-wise Recommendation: Unlike traditional methods that might recommend individual items or simple lists, DeepPage explicitly models the generation and display of a full 2D page. This involves:
- Complementary Item Generation: The framework aims to select items that work well together.
- 2D Display Strategy: A Convolutional Neural Network (CNN) is used to learn spatial correlations within the item page, optimizing how items are arranged.
- Training: The framework is trained using both online and offline procedures. For offline training, a key challenge is bridging the gap between the Actor's "proto-action" (raw item embeddings) and the "valid-action" (actual item IDs). This is handled by minimizing the difference between the proto-action and the valid-action, ensuring the Actor and Critic can learn together. The DDPG (Deep Deterministic Policy Gradient) algorithm is used for training.
Datasets
The experiments were conducted on a dataset from a real e-commerce company, collected over September 2017. This dataset comprised approximately 1 million recommendation sessions. The data was split into 70% for training/validation and 30% for testing.
Key Insights and Ideas
- Page-wise Optimization is Crucial: Treating recommendations as a 2D page with an optimized display strategy, rather than just a list, significantly improves performance. This acknowledges how users actually consume recommendations.
- Joint Generation of Items and Display: Separating item selection from display strategy is suboptimal. DeepPage's ability to jointly optimize both leads to better results.
- Real-time Preference Adaptation: Using GRUs with attention to capture dynamic user preferences is vital for effective recommendations, especially in longer sessions.
- Actor-Critic for Large Action Spaces: The Actor-Critic approach is well-suited for the massive action space of recommending entire pages of items, outperforming simpler RL methods.
- Importance of Item/Category/Feedback Embeddings: Incorporating rich representations of items, their categories, and user feedback significantly boosts recommendation accuracy.
Key Results
- Offline Performance: DeepPage outperformed several strong baselines, including Collaborative Filtering (CF), Factorization Machines (FM), GRU, DQN, and conventional DDPG, across metrics like Precision@20, Recall@20, F1-score@20, NDCG@20, and MAP. For instance, DeepPage achieved a Recall@20 of 0.3576, compared to DDPG's 0.3308.
- Online Performance: In a simulated online environment, DeepPage also showed superior performance, particularly in longer recommendation sessions, demonstrating its ability to optimize for long-term rewards. It also exhibited faster training speed compared to DDPG.
- Component Effectiveness: Ablation studies showed that each component of DeepPage (embedding layers, category/feedback information, GRUs for initial/real-time state, CNN for display strategy, and attention mechanism) contributes positively to the overall performance. Removing key components led to noticeable drops in accuracy.
Limitations and Future Work
- Temporal Complexity of Mapping: The paper acknowledges that the mapping from the Actor's "proto-action" to a "valid-action" can be computationally intensive and suggests reducing this temporal complexity as a future research direction.
- Handling Multiple Tasks: The authors propose extending the framework to collaboratively handle multiple tasks such as search, bidding, and recommendation within a single reinforcement learning system.
- Simulated Online Environment: While effective for testing, the online evaluation was conducted in a simulated environment. Real-world deployment would be the ultimate test.
Summary updated