Text-Based Interactive Recommendation via Constraint-Augmented Reinforcement Learning

Ruiyi Zhang, Tong Yu, Yilin Shen, Hongxia Jin, Changyou ChenNeurIPS 2019 · OpenReview

Interactive recommendation systems aim to find the "perfect item" by engaging in a back-and-forth dialogue with a user. While traditional systems rely on simple clicks or ratings, text-based interactive recommendation allows users to provide nuanced, natural language feedback (e.g., "I like these boots, but I want a pointed toe").

However, standard Reinforcement Learning (RL) agents struggle with this paradigm. Because RL inherently encourages "exploration" to discover better rewards, the agent often "forgets" previous constraints. For example, if a user states they dislike high heels in Round 1, a standard RL agent might suggest high heels again in Round 3 simply to explore a new part of the item space. This leads to a frustrating user experience and inefficient search.

To solve this, researchers introduced a method at NeurIPS 2019 to make interactive recommendation systems more "attentive" to user feedback. By framing natural language feedback as a set of dynamic constraints within an RL framework, the system ensures that once a user specifies a preference, the agent actually listens.

Executive Summary

The authors propose Reward Constrained Recommendation (RCR), a framework that treats natural language feedback as sequential constraints in a reinforcement learning loop. By using an adversarial discriminator to penalize recommendations that violate past user instructions, the system achieves higher success rates with fewer interactions compared to standard RL methods.

Methodology: The RCR Framework

The authors formulate the recommendation process as a Constrained Markov Decision Process (CMDP). Instead of solely maximizing a reward (finding the right item), the agent must maximize the reward subject to not violating the user's historical preferences.

Architecture of the Reward Constrained Recommendation framework

Key components of the architecture include:

  • The Discriminator ($C_\phi$): Because natural language lacks "hard" programmable rules, the authors employ a neural network discriminator. It takes the user's history of comments and the visual features of the recommended item to detect when a recommendation violates previous instructions.
  • Lagrange Relaxation: To solve the constrained optimization problem, the framework uses the Lagrange multiplier technique. This transforms user constraints into a "penalty" subtracted from the main reward. If the agent violates a constraint, the penalty ($\lambda$) increases, forcing the agent to prioritize following the user's rules.
  • Multi-Discrete Action Space: The recommender (powered by an LSTM) doesn't directly pick an item. Instead, it predicts a set of desired features. The system then retrieves the closest matching items from an image database.
  • Three-Timescale Update: The framework updates three distinct components at different frequencies: the recommender (to suggest better items), the discriminator (to improve violation detection), and the Lagrange multiplier (to balance the reward and the penalty).

Key Insights

  • Constraints as a "Dynamic Critic": Unlike standard RL where the reward function is usually fixed, the constraints in RCR are added sequentially. Every time the user speaks, the acceptable "search space" for the RL agent effectively shrinks.
  • Adversarial Generalization: By using a learned discriminator rather than hard-coded attribute matching, the system can generalize to "unseen" items where attributes might not be perfectly labeled.
  • Stability through Projection: The authors use a projection operator to keep the penalty weight within a stable range, preventing the RL training loop from diverging.

Datasets and Results

The paper evaluates the framework on two distinct tasks:

  1. Image Recommendation (UT-Zappos50K): A dataset of ~50,000 shoe images. To simulate conversation, the authors built a user simulator that generates feedback based on attribute differences between a "candidate" shoe and a "target" shoe.
  2. Constrained Text Generation (Yelp Review Dataset): An extension of the framework where the goal is to generate positive reviews while strictly avoiding negative sentiment.

The RCR framework demonstrated significant improvements over standard RL and "naive" hard-constraint baselines:

  • Efficiency: On the Zappos dataset, RCR required only 10.91 interactions on average to find the target item, compared to 26.75 for standard RL.
  • Accuracy: The Success Rate at 30 interactions (SR@30) for unseen items reached 94%, a massive jump from the 63% achieved by standard RL.
  • Violation Reduction: In the text generation task, RCR reduced the "violation rate" (generating the wrong sentiment) from 40.36% down to 10.49% while simultaneously improving the quality of the text (measured by BLEU scores).

Limitations and Future Work

While highly effective, the current framework has a few limitations. First, the recommendation results rely heavily on a user simulator; real human feedback is often noisier, more ambiguous, and less logically consistent. Second, the system currently depends on items having a predefined set of visual attributes, which might make it struggle with more abstract preferences (e.g., "I want something avant-garde"). Finally, the authors suggest that future iterations could incorporate a user’s long-term history across multiple sessions, rather than isolating memory to just the current conversation.

Summary updated

Abstract

Text-based interactive recommendation provides richer user preferences and has demonstrated advantages over traditional interactive recommender systems. However, recommendations can easily violate preferences of users from their past natural-language feedback, since the recommender needs to explore new items for further improvement. To alleviate this issue, we propose a novel constraint-augmented reinforcement learning (RL) framework to efficiently incorporate user preferences over time. Specifically, we leverage a discriminator to detect recommendations violating user historical preference, which is incorporated into the standard RL objective of maximizing expected cumulative future rewards. Our proposed framework is general and is further extended to the task of constrained text generation. Empirical results show that the proposed method yields consistent improvement relative to standard RL methods.
BibTeX
@inproceedings{zhang2019text,
  author = {Ruiyi Zhang and Tong Yu and Yilin Shen and Hongxia Jin and Changyou Chen},
  title = {{Text-Based Interactive Recommendation via Constraint-Augmented Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=LopsCV782bp},
}
Figure