User Simulator Assisted Open-ended Conversational Recommendation System

Qiusi Zhan, Xiaojie Guo, Heng Ji, Lingfei WuACL 2023

TL;DR

This paper introduces a novel framework for training open-ended conversational recommendation systems (CRS) by employing a simulated user. This approach addresses the limitations of traditional CRS that only consider immediate responses, enabling long-term reinforcement learning for improved recommendation accuracy and more informative conversations.

What Problem Does This Paper Address?

Current open-ended conversational recommendation systems (CRS) struggle with limited recommendation performance. This is largely due to their training process, which typically uses Maximum Likelihood Estimation (MLE) to predict one utterance at a time, failing to consider the long-term impact of these predictions on the overall conversation and recommendation goals. This "short-sighted" approach hinders the system's ability to learn an effective long-term recommendation policy and provide truly helpful and engaging user experiences. Traditional attribute-centric CRS, while easier to implement, lack the flexibility and interactivity of open-ended systems. The challenge lies in developing a method to train open-ended CRS to optimize for both conversational flow and accurate, informative recommendations over an extended dialogue.

Methods

The core of this work is a framework that combines a Conversational Recommendation System (CRS) with a novel User Simulator (US).

  • User Simulator (US): This US is designed to interact with the CRS using natural language, driven by specific user preferences. It comprises three key "preference-aware modules":
  • User Preference Embedding: This module captures a user's liking for a recommended item by mapping their preference vector to a continuous space, which is then incorporated into the word embeddings of the US's responses.
  • User Preference Attention: This module allows the US to actively express its short-term preferences during a dialogue, providing the CRS with more immediate and relevant information about what the user is looking for.
  • Dynamic Condensed Candidate Item Set: To manage the search space for item selection, this module dynamically updates a set of candidate items based on the user's current preferences and the last recommended item.
  • Reinforcement Learning (RL) for CRS: The framework uses the US to facilitate RL for the CRS. The CRS is fine-tuned using RL to optimize its long-term performance. This involves two novel reward signals:
  • Recommendation Reward: This reward is given based on whether the recommended item is liked by the user (positive reward) or disliked (negative reward).
  • Conversation Reward: This reward encourages the CRS to make recommendations and provide informative explanations. It includes a small positive reward for recommending an item, a positive reward for informative utterances (e.g., explaining why an item is recommended), and a negative reward for repetitive responses.

Datasets

The experiments were conducted on the REDIAL (REcommendations through DIALog) dataset. This dataset consists of dialogues collected from Amazon Mechanical Turk where workers act as recommenders and seekers discussing movie recommendations. It includes user preferences (like/dislike, seen/unseen) for mentioned movies. The paper notes that the dialogues in this dataset were used to train the US in a supervised manner, and then the CRS was fine-tuned through interaction with this US.

Key Insights and Ideas

  • The Power of Simulation for Long-Term Learning: The most significant insight is that a well-designed User Simulator can effectively enable long-term reinforcement learning for CRS. By simulating realistic user interactions, the US provides a dynamic environment that allows the CRS to learn policies that consider the future consequences of its utterances, moving beyond immediate response generation.
  • Bridging the Gap Between Conversation and Recommendation: The introduction of distinct recommendation and conversation rewards is crucial. It highlights that optimizing for both aspects simultaneously is key to a successful conversational recommender, rather than treating them as separate tasks. The conversation reward, particularly the "informative" aspect, seems to indirectly boost recommendation performance.
  • Preference-Awareness is Key for Realistic Simulation: The US's preference-aware modules are vital for generating natural and helpful user responses. Simply mimicking user preferences isn't enough; the US needs to actively express preferences and provide contextually relevant feedback to guide the CRS effectively.
  • The Informative Reward's Impact: The paper suggests that the conversation reward, especially the "informative" component, not only improves the distinctiveness of generated utterances but also indirectly enhances recommendation performance. This implies that more informative dialogue leads to better item selection.

Key Results

  • Recommendation Recall Improvement: The proposed framework significantly improves the recall of recommendations, achieving an almost 100% increase compared to the baseline NTRD model.
  • Superior Informativeness: Human evaluations demonstrate that the framework enhances the informativeness of generated utterances, making the CRS more engaging and helpful.
  • Outperforming Strong Baselines: The fine-tuned NTRD model, using the proposed framework, even outperforms the RID baseline, which leverages a pre-trained language model, in terms of recommendation performance.
  • Ablation Study Findings: Both the recommendation and conversation rewards contribute to the final performance. The conversation reward, in particular, shows a positive impact on recommendation performance.

Limitations and Future Work

  • Computational Resources: The framework requires significantly more GPU resources (double the memory) compared to training a CRS alone. This limitation prevented the authors from using large pre-trained language models (like BERT) due to their extreme memory demands.
  • Reward Design Granularity: The current informative reward is binary, treating all informative utterances equally. The authors suggest that a more nuanced reward system that can judge the level of informativeness would be beneficial for generating more persuasive recommendations.
  • Joint Optimization: The authors propose future work on jointly optimizing the CRS and the US. They believe this could further enhance the interactive qualities of the dialogue system.
  • Generalizability: Exploring the framework's generalizability to other recommendation domains is also suggested as future work.

Summary updated

BibTeX
@inproceedings{zhan2023user,
  author = {Qiusi Zhan and Xiaojie Guo and Heng Ji and Lingfei Wu},
  title = {{User Simulator Assisted Open-ended Conversational Recommendation System}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure