Text-Based Interactive Recommendation via Offline Reinforcement Learning

Ruiyi Zhang, Tong Yu, Yilin Shen, Hongxia JinAAAI 2022 · OpenReview

Abstract

Interactive recommendation with natural-language feedback can provide richer user feedback and has demonstrated advantages over traditional recommender systems. However, the classical online paradigm involves iteratively collecting experience via interaction with users, which is expensive and risky. We consider an offline interactive recommendation to exploit arbitrary experience collected by multiple unknown policies. A direct application of policy learning with such fixed experience suffers from the distribution shift. To tackle this issue, we develop a behavior-agnostic off-policy correction framework to make offline interactive recommendation possible. Specifically, we leverage the conservative Q-function to perform off-policy evaluation, which enables learning effective policies from fixed datasets without further interactions. Empirical results on the simulator derived from real-world datasets demonstrate the effectiveness of our proposed offline training framework.

Introduction

Interactive recommendation systems, such as digital personal assistants, allow users to refine suggestions through natural language feedback (e.g., "I want these shoes, but with a lower heel"). Traditionally, these systems rely on Online Reinforcement Learning (RL), where an agent learns by interacting with real users in real-time. However, online exploration is often impractical due to high costs, slow convergence, and the risk of frustrating users with irrelevant recommendations during the learning phase.

Comparison between online and offline training paradigms for interactive recommendation.

Offline RL offers a compelling alternative by learning from fixed datasets of past interactions. The primary challenge in this setting is "distribution shift." When training data is aggregated from diverse, unknown sources—typical in privacy-protected environments—standard RL algorithms often overestimate the value of unfamiliar actions. This leads to poor performance upon deployment. This paper addresses these challenges by proposing a behavior-agnostic off-policy correction framework.

Methodology

The proposed framework enables effective policy learning from static interaction logs without requiring a known behavior policy.

Conservative Q-Learning (CQL)

To mitigate overestimation bias caused by distribution shift, the system employs Conservative Q-Learning. Unlike standard Q-learning, CQL penalizes Q-values for state-action pairs that are underrepresented in the training data. This ensures the model remains conservative and operates within the "safe" boundaries of its observed experience.

Multimodal State Tracking

Interactive recommendation requires processing both visual and textual inputs. The architecture includes a Multimodal Encoder that utilizes a ResNet for images and an LSTM for natural language feedback. A State Tracker maintains a latent representation of user preferences across multiple conversational turns.

The proposed architecture featuring multimodal encoding and a recommender with reward correction.

Iterative Offline Training

The authors implement an iterative loop where the model is trained offline, deployed to a simulator to collect new data, and then retrained. This allows for gradual improvement of the experience base without live human intervention during core training phases.

Experimental Setup

The framework was evaluated using the UT-Zappos50K dataset, which contains over 50,000 shoe images with detailed attributes. Due to the scarcity of real-world interactive logs, a User Simulator was developed. This simulator generates natural language feedback based on attribute differences between a target item and the current recommendation (e.g., "I want boots" if shown a sneaker).

Results and Discussion

The Iterative Offline method demonstrated significant improvements over traditional baselines:

  • Success Rate (SR@10): The model achieved a 78% success rate, outperforming Imitation Learning (69%) and standard Offline Learning (72%).
  • Efficiency: The average Number of Interactions (NI) required to find the target item dropped to 10.31, compared to 13.88 for the baseline behavior.
  • Performance Gap: The offline model's performance approached the on-policy "upper bound" of 84%, which assumes unlimited live user interaction.

A key insight is that this approach can learn from "failure" logs. By analyzing recommendations that received negative feedback, the model learns to avoid similar mistakes. Furthermore, the framework is behavior-agnostic, making it suitable for real-world applications where data is collected by millions of different local policies without central coordination.

Conclusion

This work demonstrates that text-based interactive recommendation can be effectively trained using offline RL, bypassing the risks of online exploration. While the current study uses simulated language, future work will focus on handling the linguistic diversity of real human conversation and validating the framework on massive real-world interaction logs.

Summary updated

Abstract

Interactive recommendation with natural-language feedback can provide richer user feedback and has demonstrated advantages over traditional recommender systems. However, the classical online paradigm involves iteratively collecting experience via interaction with users, which is expensive and risky. We consider an offline interactive recommendation to exploit arbitrary experience collected by multiple unknown policies. A direct application of policy learning with such fixed experience suffers from the distribution shift. To tackle this issue, we develop a behavior-agnostic off-policy correction framework to make offline interactive recommendation possible. Specifically, we leverage the conservative Q-function to perform off-policy evaluation, which enables learning effective policies from fixed datasets without further interactions. Empirical results on the simulator derived from real-world datasets demonstrate the effectiveness of our proposed offline training framework.
BibTeX
@inproceedings{zhang2022text,
  author = {Ruiyi Zhang and Tong Yu and Yilin Shen and Hongxia Jin},
  title = {{Text-Based Interactive Recommendation via Offline Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=gYUqbPtWx6},
}
Figure