Fashion Image Retrieval With Text Feedback by Additive Attention Compositional Learning

Yuxin Tian, Shawn Newsam, Kofi BoakyeWACV 2023

TL;DR

This paper introduces Additive Attention Compositional Learning (AACL), a novel framework for fashion image retrieval that uses text feedback to modify a source image. AACL employs a unique additive attention mechanism within a transformer architecture to effectively fuse image and text information, achieving state-of-the-art results on multiple fashion retrieval benchmarks.

What Problem Does This Paper Address?

Traditional image retrieval often struggles to incorporate user intent for refining search results. While query images and keywords are common, users often want to specify how an image should be modified. This paper tackles the challenge of fashion image retrieval with text feedback, where a user provides a source image and natural language instructions to find a target image that resembles the source but incorporates specific changes (e.g., "dress is blue with a higher neckline"). This requires a model to understand both visual content and linguistic descriptions, and to compose these into a coherent query that can identify the desired target image.

Methods

The core of AACL is its Additive Attention Composition Module. This module is designed to be plugged into existing deep neural networks and leverages a transformer-based architecture.

  • Image and Text Encoders: The framework uses a Swin Transformer for image encoding, capturing multi-level visual features by concatenating tokens from the final two stages. For text encoding, it employs DistilBERT.
  • Additive Attention Composition Module: This is the novel component. Instead of standard dot-product attention, AACL uses an additive attention mechanism. This mechanism learns a context vector that selectively modifies the image representation based on the text feedback. The authors argue this is more efficient and can achieve higher accuracy than multiplicative attention, especially for longer sequences.
  • Composition Block: The module is built with multiple composition blocks, each containing an additive self-attention layer, a linear layer, and a feed-forward network. This allows for a single-stage fusion of image and text features.
  • Deep Metric Learning: The model is trained using a batch-based classification loss to push similar image-text pairs closer in the latent space and dissimilar pairs further apart.

Datasets

The paper evaluates AACL on three datasets:

  • FashionIQ: A natural language-based interactive fashion product retrieval dataset with images from Amazon.com, covering Dresses, Tops&Tees, and Shirts.
  • Fashion200k: A large-scale dataset with over 200k fashion images from online shopping websites, covering Dresses, Jackets, Pants, Skirts, and Tops.
  • Shopping100k (modified): A new, challenging benchmark derived from the original Shopping100k dataset. This dataset contains more diverse fashion items in isolation and features 11 high-level apparel categories. The authors created this dataset to provide a more challenging benchmark for the research community.

Key Insights and Ideas

  • Additive Attention for Efficiency and Accuracy: The paper highlights that additive attention, compared to dot-product attention, offers a more efficient and potentially more accurate way to fuse image and text information, especially in the context of complex modifications.
  • Selective Modification of Image Features: The core idea of AACL is to selectively modify image features based on text feedback, preserving relevant visual aspects while transforming others. This is crucial for tasks like "replace color" or "change neckline."
  • Multi-level Image Representation: Using features from multiple stages of the Swin Transformer (specifically Stages 3 and 4) proved more effective than using only the final stage, suggesting that lower-level visual details are important for fine-grained fashion retrieval.
  • Compositional Learning is Key: The success of AACL underscores the importance of a dedicated module for composing image and text information, rather than relying on simpler fusion methods.

Key Results

AACL consistently achieves state-of-the-art results across all three evaluated datasets:

  • FashionIQ: AACL outperforms existing methods by a significant margin, including those that use more complex approaches. When compared head-to-head with similar encoder setups, AACL surpasses strong baselines like TIRG, RTIC, and MAAF by 1.41% to 3.42% in average R@10 and R@50 scores.
  • Fashion200k: AACL achieves compelling results, outperforming the best competitor (MAAF) by a relative margin of 9.4% for R@1.
  • Shopping100k: AACL again demonstrates superior performance, outperforming SOTA baselines. Qualitative results show AACL can handle complex text descriptions and jointly comprehend global appearance and local fine-grained details.

Qualitative examples demonstrate AACL's ability to retrieve images that accurately reflect the specified modifications, such as changing colors, necklines, and sleeve lengths, while maintaining the overall style of the original image.

Limitations and Future Work

  • Dataset Limitations: The retrieved images are inherently limited by the content present in the target datasets. If a desired modification doesn't exist in the dataset, AACL cannot fulfill it.
  • Attention Visualization: Current attention visualization methods are primarily designed for dot-product attention and are not directly compatible with AACL's additive attention module, leading to a potentially less precise visualization approach.
  • Data Annotation: The paper notes that template-based relative caption generation, while widely used, is not as accurate or diverse as human annotations, which can impact training data quality.

Future work suggested by the authors includes:

  • Leveraging recent advances in image generation to create realistic and desired images based on query image-text pairs.
  • Developing automated relative captioning methods to generate more natural language text modifiers and reduce query text noise.

Summary updated

BibTeX
@inproceedings{tian2023fashion,
  author = {Yuxin Tian and Shawn Newsam and Kofi Boakye},
  title = {{Fashion Image Retrieval With Text Feedback by Additive Attention Compositional Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}

Loading related content...

Figure