Multimodal Recommendation Dialog with Subjective Preference: A New Challenge and Benchmark

Yuxing Long, Binyuan Hui, Caixia Yuan, Fei Huang, Yongbin Li, Xiaojie Wang

Introduction

In the realm of task-oriented dialog (TOD) systems, multimodal interaction has primarily focused on objective "referring expressions"—identifying items based on concrete physical attributes like color, position, or category. However, real-world shopping experiences are rarely so clinical. Customers often express needs through subjective preferences, such as "something for a summer party" or "clothes that feel calm." Bridging this gap requires AI agents to navigate a complex "many-to-many" mapping between vague human desires and specific product inventories. To address this, we introduce SURE (Multimodal Recommendation Dialog with SUbjective PREference), a large-scale benchmark designed to push the boundaries of multimodal recommendation.

The SURE Dataset: Capturing Subjective Nuance

The SURE dataset comprises 12,180 dialogs and over 223,000 utterances, spanning the fashion and furniture domains. Unlike previous datasets like SIMMC 2.0, which rely on objective labels, SURE focuses on the interplay between subjective language and expert sales strategies.

Dataset Construction

The creation of SURE followed a rigorous two-phase pipeline:

  1. Expert-Informed Simulation: Researchers surveyed 238 sales professionals to identify eight core "Salesperson Acts," such as Ask Preference, Refer Region, and Guess Attribute Value. These strategies were used to simulate logical dialog flows within 1,566 virtual store scenes.
  2. Human Paraphrasing: To move beyond robotic templates, human annotators replaced simulated concepts with diverse subjective expressions. For instance, a "warm color" might be described as "the color of passion" or "a hue for a happy activity."

This process resulted in over 3,000 unique subjective preferences, highlighting the inherent diversity and ambiguity of human shopping language.

The Challenge of Subjectivity

The core difficulty of the SURE benchmark lies in the "Subjectivity Gap." An AI agent must perform a two-step translation:

  • Semantic Mapping: Translating a subjective term (e.g., "lively pattern") into a set of potential categorization concepts (e.g., floral, leopard, or star designs).
  • Inventory Filtering: Applying these concepts to the current store inventory to find matching items.

This mapping is non-deterministic; "red" might satisfy a request for "passion," "energy," or simply "a summer vibe." Conversely, a single subjective request can map to multiple distinct physical attributes.

Benchmark Tasks and Baselines

To evaluate progress, the authors established three key tasks:

  1. Subjective Preference Disambiguation (SPD): Predicting concrete attribute values from subjective text.
  2. Referred Region Understanding (RRU): Identifying items within specific spatial regions (e.g., "the back right rack") based on visual scene snapshots.
  3. Multimodal Recommendation (MR): Predicting the next salesperson act and generating a natural language response.

The proposed Multimodal Recommendation Agent (MRA) serves as a baseline. Built on the OFA (One-For-All) architecture, MRA processes dialog history, item metadata, and visual scene patches within a unified Transformer framework.

Key Findings

Experiments reveal significant room for improvement across the board:

  • Low Performance on Subjectivity: The MRA baseline achieved an F1 score of only ~37.5% on the SPD task, illustrating the difficulty of grounding vague descriptors.
  • Spatial Reasoning Struggles: On the RRU task, the model scored ~14.8% F1, indicating that grounding spatial language in complex, crowded visual scenes remains a major hurdle.
  • Multimodal Necessity: Ablation studies confirmed that removing either visual scene snapshots or item metadata led to a sharp decline in performance, proving that the SURE tasks require deep multimodal integration.

Conclusion and Future Work

SURE introduces a more realistic and challenging paradigm for multimodal recommendation. While the MRA baseline provides a starting point, the low scores across all tasks highlight the need for more sophisticated reasoning modules capable of handling spatial relations and the inherent ambiguity of subjective language. Future work will focus on expanding the dataset to multiple languages and exploring specialized pre-training techniques to better bridge the subjectivity gap.

Summary updated

Abstract

Existing multimodal task-oriented dialog data fails to demonstrate the diverse expressions of user subjective preferences and recommendation acts in the real-life shopping scenario. This paper introduces a new dataset SURE (Multimodal Recommendation Dialog with SUbjective PREference), which contains 12K shopping dialogs in complex store scenes. The data is built in two phases with human annotations to ensure quality and diversity. SURE is well-annotated with subjective preferences and recommendation acts proposed by sales experts. A comprehensive analysis is given to reveal the distinguishing features of SURE. Three benchmark tasks are then proposed on the data to evaluate the capability of multimodal recommendation agents. Based on the SURE, we propose a baseline model, powered by a state-of-the-art multimodal model, for these tasks.
BibTeX
@inproceedings{long2023multimodal,
  author = {Yuxing Long and Binyuan Hui and Caixia Yuan and Fei Huang and Yongbin Li and Xiaojie Wang},
  title = {{Multimodal Recommendation Dialog with Subjective Preference: A New Challenge and Benchmark}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.217},
}
Figure