Abstract
With recent advances in multimodal interactive recommendations, users can express preferences through natural language feedback on item images. However, existing systems often require manual selection (e.g., clicking or touching) of the items being commented on, which prevents a truly "hands-free" experience. We propose a hands-free visual dialog recommender system that identifies discussed items and predicts desired attributes directly from natural language feedback. Our system utilizes a novel Sleeping Pairwise Ranking (SPR) bandit algorithm, which constrains exploration to relevant items and leverages relative preferences for sample-efficient learning. Empirical results demonstrate that our system is three times more effective at finding target items than traditional interactive recommenders after only a few rounds of interaction.
Introduction
Most modern interactive recommendation systems rely on a "click-to-clarify" loop. While these systems allow for natural language feedback (e.g., "I want something more formal than these shoes"), they typically require the user to specify which item they are referencing. This requirement creates significant hurdles for accessibility and usability. In scenarios where a user's hands are occupied—such as cooking, driving, or caring for a child—manual interaction is impossible. Furthermore, requiring a click introduces ambiguity; if a user says "I prefer black shoes" while looking at a list, the system must know which items triggered that preference to update its model effectively.
We address these challenges by developing a system that can disambiguate which items are being discussed and learn user preferences efficiently from very few interactions. By moving away from the click-based paradigm, we enable a more natural, conversational interface for item discovery.
Proposed Framework
The proposed architecture consists of three main components that work in concert to translate spoken comments into refined recommendations.
1. Item Identifier
This module predicts which items in the current recommended list are the subject of the user's feedback. It utilizes a multimodal approach, employing a ResNet101 image encoder and a CNN-based text encoder. By comparing the visual features of the recommended items with the semantic content of the user's comment, the identifier determines the relevance of each item to the ongoing dialog.
2. Visual Dialog Encoder
Once the relevant items are identified, the Visual Dialog Encoder infers the user's "desired attributes." It maps the relationship between the current item and the user's feedback into a latent space that represents the user's target preference. This allows the system to understand not just what the user doesn't like, but the direction in which they want the recommendations to move.
3. Sleeping Pairwise Ranking (SPR) Bandit
The SPR Bandit serves as the system's decision engine, balancing the exploration of new items with the exploitation of known preferences using Thompson Sampling. It introduces two critical innovations:
- Constrained Exploration: The system treats the task as a "sleeping bandit" problem. Only items that roughly match the inferred desired attributes are considered "awake" or available for recommendation. This prevents the system from wasting rounds on irrelevant items, significantly speeding up convergence.
- Pairwise Logistic Loss: To overcome the "cold start" problem where data is sparse, the system derives additional training samples from relative preferences. If a user expresses a preference for "black flats" over the current "brown boots," the system treats this relative difference as a strong learning signal. The model is jointly updated using both traditional logistic loss and this new pairwise loss.
Experimental Evaluation
The authors evaluated the system using a large-scale Footwear dataset containing over 14,000 images. 10,000 images were used for pre-training the multimodal encoders, while 4,658 were reserved for online recommendation testing. To simulate real-world usage, a "relative captioner" model acted as a surrogate for human users, providing feedback based on the differences between recommended items and a hidden target item.
The results highlight the efficiency of the SPR Bandit. After just 10 rounds of interaction, the system achieved three times the average cumulative reward of standard baselines like CascadeUCB1 and CascadeLinTS. In a user study involving 400 runs, participants found an average of 11.8 desired items within 30 rounds using the SPR system, compared to only 4.7 items with traditional baselines. Furthermore, users reached their first desired item in an average of 6 rounds, nearly twice as fast as the baseline's 13 rounds.
Conclusion and Future Work
The hands-free visual dialog recommender represents a significant step toward more accessible and natural AI interfaces. By combining multimodal disambiguation with constrained bandit exploration, the system achieves high sample efficiency without requiring manual user input. Future research will focus on improving robustness to Automatic Speech Recognition (ASR) errors in noisy environments and handling "preference drift," where a user's goals evolve during the interaction process.
Summary updated