Introduction
Reinforcement Learning (RL) has long been viewed as a promising framework for recommendation systems because it optimizes for long-term user engagement rather than immediate clicks. However, the transition from theoretical RL to production environments is fraught with challenges, particularly regarding offline training. Traditional RL agents learn through exploration—a process that involves making potentially poor recommendations to discover optimal strategies. In real-world e-commerce, such exploration is prohibitively expensive, leading to a reliance on offline historical data.
To address these hurdles, researchers have introduced Prompt-Based Reinforcement Learning (PRL). This new paradigm reframes the recommendation problem by treating historical interaction data as a "knowledge base" and using desired rewards as a prompt to directly infer the best action. By flipping the traditional RL objective, PRL significantly improves performance and stability in offline training scenarios.
The Offline Training Challenge
Standard RL algorithms, such as Q-learning, typically attempt to map a state and an action to an expected reward: (state, action) → expected reward. When trained offline using data collected by older algorithms, these models suffer from "distribution shift." The agent struggles to accurately estimate the value of actions it hasn't seen frequently in the historical logs, often leading to sub-optimal performance or high variance.
Furthermore, traditional offline RL is susceptible to the "deadly triad" of instability: off-policy learning, function approximation, and bootstrapping. These factors combined make it difficult to deploy robust RL-based recommenders in complex, high-stakes environments like retail.
Methodology: The PRL Paradigm
PRL departs from value estimation and instead adopts a supervised learning approach inspired by the prompting trends in Large Language Models (LLMs).
1. The Conceptual Flip
Instead of predicting rewards, PRL maps a state and a target reward to an action: (state, reward) → action. During training, the model examines historical sessions where the state (past interactions), the action (the item recommended), and the actual cumulative reward are all known. It learns to answer the question: "Given this specific user history and this eventual reward outcome, which item was chosen?"
2. Model Architecture
The PRL architecture consists of three primary components:
- State Encoding: A sequential model (such as a GRU, CNN, or SASRec) encodes the user's interaction history into a representative state.
- Prompt Construction: This state is combined with a reward embedding and a step embedding, creating a comprehensive prompt for the model.
- Self-Attentive Block: A transformer-style self-attention layer processes the relationship between the state and the prompted reward. This allows the model to "reason" about how different reward targets correlate with specific item choices.
3. Inference via Prompting
At deployment, the developer "prompts" the model by inputting a high target reward (e.g., a value higher than the average session reward). The model then infers the action most likely to achieve that specific goal based on the patterns it identified in the training data. This effectively treats RL as an information retrieval task where the desired reward is the search query.
Experimental Evaluation
The authors evaluated PRL on two large-scale e-commerce datasets:
- Challenge15: Derived from the RecSys Challenge 2015, featuring user click and purchase sequences.
- RetailRocket: Real-world data involving "view" and "add-to-cart" behaviors.
Both datasets were pre-processed to focus on meaningful session lengths, resulting in approximately 200,000 sequences for training and testing respectively.
Key Results
PRL consistently outperformed state-of-the-art offline RL baselines, including Self-Supervised Q-Learning (SQN) and Self-Supervised Actor-Critic (SAC). On the RetailRocket dataset, PRL achieved a significant boost in Hit Ratio (HR@5) for purchases, rising from a base GRU performance of 0.4608 to 0.5486. Additionally, improvements in NDCG indicated that PRL is superior at ranking high-value items at the top of the recommendation list.
Ablation studies confirmed that the Self-Attentive Block is vital; replacing it with simpler architectures like MLPs led to a noticeable performance drop, suggesting that the complex interaction between the state and the reward prompt is key to the model's success.
Discussion and Future Work
PRL offers several advantages over traditional methods. By avoiding recursive value estimation, it bypasses the instability of the "deadly triad." It also introduces a level of controllability; developers can theoretically tune the recommender's behavior at inference time—prompting for higher diversity or specific conversion goals—without retraining the model.
However, several avenues for future research remain:
- Optimal Prompting: Determining the exact reward value to prompt with during inference is currently done through heuristics; finding an automated "optimal" prompt is an open question.
- Temporal Dynamics: The current model focuses on session-based rewards. Future iterations could better handle long-term delayed rewards that occur days after an initial interaction.
- Online Validation: While offline results are compelling, live production testing is necessary to observe how PRL handles dynamic user feedback in real-time.
Conclusion
Prompt-Based Reinforcement Learning represents a significant shift in how we approach recommendation tasks. By reframing RL as a prompted retrieval problem, PRL provides a more stable, scalable, and controllable alternative to traditional offline RL, paving the way for more effective long-term optimization in digital marketplaces.
Summary updated