Abstract
Conversational interfaces for the detail-oriented retail fashion domain are more natural, expressive, and user friendly than classical keyword-based search interfaces. In this paper, we introduce the Fashion IQ dataset to support and advance research on interactive fashion image retrieval. Fashion IQ is the first fashion dataset to provide human-generated captions that distinguish similar pairs of garment images together with side-information consisting of real-world product descriptions and derived visual attribute labels for these images. We provide a detailed analysis of the characteristics of the Fashion IQ data, and present a transformer-based user simulator and interactive image retriever that can seamlessly integrate visual attributes with image features, user feedback, and dialog history, leading to improved performance over the state of the art in dialog-based image retrieval. We believe that our dataset will encourage further work on developing more natural and real-world applicable conversational shopping assistants.
The Shift to Dialog-Based Search
Searching for the "perfect" outfit is notoriously difficult for traditional search engines. Most current systems rely on "Classical Fashion Search," where a user clicks through a predefined ontology of filters like color, length, or sleeve type. This process is often cumbersome and fails to capture the nuance of a user's specific desires.
The real challenge lies in interactive image retrieval: how can a system engage in a multimodal conversation, using natural language feedback to refine search results over multiple turns? Until now, progress was stalled by a lack of high-quality data that paired images with the kind of comparative feedback humans actually give, such as "I want something more formal than this, with shorter sleeves."
The Fashion IQ Dataset
To address this gap, the researchers introduced Fashion IQ, a large-scale dataset uniquely positioned among fashion benchmarks. It contains approximately 77,000 images across three categories: Dresses, Shirts, and Tops & Tees.
Key features of Fashion IQ include:
- Relative Captions: Over 60,000 human-generated captions that describe the differences between similar items.
- Side Information: Real-world product descriptions and 1,000 distinct attribute labels (texture, fabric, shape, etc.) mined from web metadata.
- Relational Focus: Unlike datasets that focus on absolute labels (e.g., "this is a blue dress"), Fashion IQ focuses on the relationship between items, which is critical for conversational AI.
Methodology: Transformers for Retrieval
The researchers developed a novel architecture using a Multimodal Transformer for the interactive retriever. This model ingests three distinct inputs: visual features (raw pixel data extracted via EfficientNet), visual attributes (high-level labels derived from product descriptions), and natural language feedback from the user.
By using self-attention, the Transformer weighs the importance of the dialog history and specific visual cues to "shift" the search query toward the target image in the feature space. To train this system, the authors also built a Transformer-based User Simulator that generates relative captions describing the differences between a reference and a target image.
Key Results and Insights
The proposed Transformer-based models set a new state-of-the-art (SOTA) on the Fashion IQ benchmark. Quantitative analysis showed that the model's ability to narrow down search results improves significantly as the conversation progresses. For instance, in the "Dresses" category, the model achieved a Recall@10 of 13.39% on the first turn, which jumped to 46.28% by the fifth turn of the dialog.
Furthermore, the inclusion of visual attributes as "side information" proved vital. In the "Tops & Tees" category, adding these attributes increased the Recall@10 from 33.52% to 35.94% by the final dialog turn. Qualitatively, the model demonstrates a sophisticated understanding of user intent; when a user requests a "little more red accent," the retriever identifies items that maintain the style of the current selection while incorporating the specific requested color change.
Conclusion
While Fashion IQ represents a major step forward, challenges remain regarding the cost of human annotation and the gap between simulators and real-world human behavior. Future work will focus on developing conversational shopping assistants capable of handling more complex, multi-turn reasoning and offering stylistic advice rather than just retrieving images.
Summary updated