Towards Hands-Free Visual Dialog Interactive Recommendation

Tong Yu, Yilin Shen, Hongxia JinAAAI 2020 · OpenReview

Abstract

With recent advances in multimodal interactive recommendations, users can express preferences through natural language feedback on item images. However, existing systems often require manual selection (e.g., clicking or touching) of the items being commented on, which prevents a truly "hands-free" experience. We propose a hands-free visual dialog recommender system that identifies discussed items and predicts desired attributes directly from natural language feedback. Our system utilizes a novel Sleeping Pairwise Ranking (SPR) bandit algorithm, which constrains exploration to relevant items and leverages relative preferences for sample-efficient learning. Empirical results demonstrate that our system is three times more effective at finding target items than traditional interactive recommenders after only a few rounds of interaction.

Introduction

Most modern interactive recommendation systems rely on a "click-to-clarify" loop. While these systems allow for natural language feedback (e.g., "I want something more formal than these shoes"), they typically require the user to specify which item they are referencing. This requirement creates significant hurdles for accessibility and usability. In scenarios where a user's hands are occupied—such as cooking, driving, or caring for a child—manual interaction is impossible. Furthermore, requiring a click introduces ambiguity; if a user says "I prefer black shoes" while looking at a list, the system must know which items triggered that preference to update its model effectively.

Example of a multi-round hands-free visual dialog interaction.

We address these challenges by developing a system that can disambiguate which items are being discussed and learn user preferences efficiently from very few interactions. By moving away from the click-based paradigm, we enable a more natural, conversational interface for item discovery.

Proposed Framework

The proposed architecture consists of three main components that work in concert to translate spoken comments into refined recommendations.

1. Item Identifier

This module predicts which items in the current recommended list are the subject of the user's feedback. It utilizes a multimodal approach, employing a ResNet101 image encoder and a CNN-based text encoder. By comparing the visual features of the recommended items with the semantic content of the user's comment, the identifier determines the relevance of each item to the ongoing dialog.

2. Visual Dialog Encoder

Once the relevant items are identified, the Visual Dialog Encoder infers the user's "desired attributes." It maps the relationship between the current item and the user's feedback into a latent space that represents the user's target preference. This allows the system to understand not just what the user doesn't like, but the direction in which they want the recommendations to move.

3. Sleeping Pairwise Ranking (SPR) Bandit

The SPR Bandit serves as the system's decision engine, balancing the exploration of new items with the exploitation of known preferences using Thompson Sampling. It introduces two critical innovations:

  • Constrained Exploration: The system treats the task as a "sleeping bandit" problem. Only items that roughly match the inferred desired attributes are considered "awake" or available for recommendation. This prevents the system from wasting rounds on irrelevant items, significantly speeding up convergence.
  • Pairwise Logistic Loss: To overcome the "cold start" problem where data is sparse, the system derives additional training samples from relative preferences. If a user expresses a preference for "black flats" over the current "brown boots," the system treats this relative difference as a strong learning signal. The model is jointly updated using both traditional logistic loss and this new pairwise loss.
The three-component architecture of the hands-free recommender system.

Experimental Evaluation

The authors evaluated the system using a large-scale Footwear dataset containing over 14,000 images. 10,000 images were used for pre-training the multimodal encoders, while 4,658 were reserved for online recommendation testing. To simulate real-world usage, a "relative captioner" model acted as a surrogate for human users, providing feedback based on the differences between recommended items and a hidden target item.

The results highlight the efficiency of the SPR Bandit. After just 10 rounds of interaction, the system achieved three times the average cumulative reward of standard baselines like CascadeUCB1 and CascadeLinTS. In a user study involving 400 runs, participants found an average of 11.8 desired items within 30 rounds using the SPR system, compared to only 4.7 items with traditional baselines. Furthermore, users reached their first desired item in an average of 6 rounds, nearly twice as fast as the baseline's 13 rounds.

Conclusion and Future Work

The hands-free visual dialog recommender represents a significant step toward more accessible and natural AI interfaces. By combining multimodal disambiguation with constrained bandit exploration, the system achieves high sample efficiency without requiring manual user input. Future research will focus on improving robustness to Automatic Speech Recognition (ASR) errors in noisy environments and handling "preference drift," where a user's goals evolve during the interaction process.

Summary updated

Abstract

With the recent advances of multimodal interactive recommendations, the users are able to express their preference by natural language feedback to the item images, to find the desired items. However, the existing systems either retrieve only one item or require the user to specify (e.g., by click or touch) the commented items from a list of recommendations in each user interaction. As a result, the users are not hands-free and the recommendations may be impractical. We propose a hands-free visual dialog recommender system to interactively recommend a list of items. At each time, the system shows a list of items with visual appearance. The user can comment on the list in natural language, to describe the desired features they further want. With these multimodal data, the system chooses another list of items to recommend. To understand the user preference from these multimodal data, we develop neural network models which identify the described items among the list and further predict the desired attributes. To achieve efficient interactive recommendations, we leverage the inferred user preference and further develop a novel bandit algorithm. Specifically, to avoid the system exploring more than needed, the desired attributes are utilized to reduce the exploration space. More importantly, to achieve sample efficient learning in this hands-free setting, we derive additional samples from the user's relative preference expressed in natural language and design a pairwise logistic loss in bandit learning. Our bandit model is jointly updated by the pairwise logistic loss on the additional samples derived from natural language feedback and the traditional logistic loss. The empirical results show that the probability of finding the desired items by our system is about 3 times as high as that by the traditional interactive recommenders, after a few user interactions.
BibTeX
@inproceedings{yu2020towards,
  author = {Tong Yu and Yilin Shen and Hongxia Jin},
  title = {{Towards Hands-Free Visual Dialog Interactive Recommendation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=IaeRC1nmqy},
}
Figure