The Fashion IQ Dataset: Retrieving Images by Combining Side Information and Relative Natural Language Feedback

Xiaoxiao Guo, Hui Wu, Yupeng Gao, Steven Rennie, Rogério Schmidt Feris

TL;DR

This paper introduces Fashion IQ, a novel dataset for interactive fashion image retrieval that combines human-generated relative captions with rich product side information (attributes, descriptions). They also present a multimodal transformer-based system that leverages these modalities to achieve state-of-the-art performance in both user simulation and dialog-based image retrieval, paving the way for more natural and effective conversational shopping assistants.

What Problem Does This Paper Address?

Traditional fashion search relies on keyword-based or attribute-based filtering, which can be cumbersome and lack expressiveness. While natural language feedback has shown promise, existing systems often struggle to effectively integrate diverse information sources like visual attributes and user feedback within a conversational context. The authors aim to bridge this gap by creating a dataset that facilitates research into systems that can understand and utilize both explicit product information and nuanced, natural language user interactions for more intuitive and accurate image retrieval.

Methods

The core of the paper's contribution lies in two main areas:

  • Fashion IQ Dataset: This novel dataset is designed to support interactive fashion image retrieval. It includes:
  • Human-generated relative captions: These captions describe the subtle differences between pairs of similar garment images, offering a richer form of feedback than simple keywords.
  • Side Information: This includes real-world product descriptions and derived visual attribute labels, providing valuable context that is often available in e-commerce settings.
  • Multimodal Transformer Models: The authors propose two transformer-based models:
  • Relative Captioning Transformer: This model acts as a user simulator, generating relative captions by attending to both image features and attribute information.
  • Dialog-based Image Retrieval Transformer: This model integrates image features, fashion attributes, and user textual feedback (from the simulator or a real user) to iteratively refine search results. The transformer architecture allows for effective fusion of these multimodal inputs.

Datasets

  • Fashion IQ Dataset (New Benchmark): This is the primary dataset introduced in the paper. It's unique in its combination of:
  • Human-written relative captions for image pairs.
  • Associated real-world product descriptions and attribute labels.
  • It covers three fashion categories: Dresses, Shirts, and Tops&Tees.

The dataset was constructed by sourcing images from a product review dataset and collecting annotations through crowdsourcing.

Key Insights and Ideas

  • Relative Captions are More Expressive: The analysis of Fashion IQ reveals that relative captions offer a more flexible and expressive way for users to articulate their search intent compared to traditional attribute labels.
  • Side Information is Crucial: The paper strongly suggests that combining visual attributes with natural language feedback significantly boosts performance in interactive image retrieval. Attributes provide a structured way to highlight specific garment characteristics that might be missed by pure visual or textual analysis.
  • Transformer Architecture Excels: The transformer's self-attention mechanism proves highly effective at integrating diverse modalities (images, attributes, text) and historical dialog context, leading to superior performance over previous RNN-based approaches.
  • User Simulation is Key for Training: The ability to simulate user feedback using a relative captioning model allows for efficient training of dialog-based retrieval systems without requiring constant human interaction, which is crucial for scalability.

Key Results

  • Relative Captioning: The proposed transformer model for relative captioning outperforms the state-of-the-art RNN-based baseline by a significant margin across standard captioning metrics (BLEU-4, Rouge-L, CIDER, SPICE). Incorporating attributes further improves performance.
  • Dialog-based Image Retrieval: The multimodal transformer-based dialog retriever, when paired with the transformer-based user simulator, achieves state-of-the-art performance. For instance, in dialog turn 5, their model (Retriever (T) + User (T)) achieves a ranking percentile (P) of 98.39 for Dresses, significantly outperforming the RNN-based baseline (97.96). The inclusion of attributes consistently leads to better retrieval results.
  • Single-turn Retrieval: The multimodal retrieval model (Image + attributes + relative captions) also shows superior performance compared to single-modality baselines and the RNN-based approach.

Limitations and Future Work

  • Dataset Cost: While the human-generated captions are high-quality, their collection is costly. The paper acknowledges this but emphasizes the benefit for building robust systems.
  • Model Training: The paper mentions that the user simulator and retriever are trained independently, and while this is a practical approach, future work could explore end-to-end training for potentially even better synergy.
  • Future Directions: The authors believe their dataset and framework will serve as stepping stones for developing more effective interactive image retrieval systems. They suggest further exploration of how to best leverage multimodal information and natural language interfaces for conversational shopping assistants.

Summary updated

Abstract

Conversational interfaces for the detail-oriented retail fashion domain are more natural, expressive, and user friendly than classical keyword-based search interfaces. In this paper, we introduce the Fashion IQ dataset to support and advance research on interactive fashion image retrieval. Fashion IQ is the first fashion dataset to provide human-generated captions that distinguish similar pairs of garment images together with side-information consisting of real-world product descriptions and derived visual attribute labels for these images. We provide a detailed analysis of the characteristics of the Fashion IQ data, and present a transformer-based user simulator and interactive image retriever that can seamlessly integrate visual attributes with image features, user feedback, and dialog history, leading to improved performance over the state of the art in dialog-based image retrieval. We believe that our dataset will encourage further work on developing more natural and real-world applicable conversational shopping assistants.
BibTeX
@misc{guo2019fashion,
  author = {Xiaoxiao Guo and Hui Wu and Yupeng Gao and Steven Rennie and Rogério Schmidt Feris},
  title = {{The Fashion IQ Dataset: Retrieving Images by Combining Side Information and Relative Natural Language Feedback}},
  year = {2019},
  eprint = {1905.12794},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.12794},
}
Figure