Compositional Learning of Image-Text Query for Image Retrieval

Muhammad Umer Anwaar, Egor Labintcev, Martin KleinsteuberWACV 2021

TL;DR

This paper introduces ComposeAE, a novel autoencoder-based method for image retrieval using both an image and a text query that specifies desired modifications. ComposeAE learns to compose these modalities by mapping them into a complex space where image and text features are treated as rotations, outperforming existing state-of-the-art methods on several benchmark datasets.

What Problem Does This Paper Address?

Traditional image retrieval systems typically rely on either an image or a text query. However, users often have a more nuanced request, like finding an item similar to one they've seen but with specific modifications (e.g., a dress like a friend's, but in a different color and with a sash). This paper addresses the challenge of multi-modal image retrieval where the query consists of a source image and a natural language description of desired changes. Existing methods, like TIRG, struggle with longer, more descriptive text queries and may not effectively capture the compositional nature of such requests.

Methods

The core of ComposeAE is its innovative approach to composing image and text query features. Instead of simply concatenating or gating features in a real space, ComposeAE maps both the source image features (extracted via a CNN like ResNet-17) and the text query features (extracted via BERT) into a complex space. In this complex space, the relationship between the source image and the target image is modeled as a rotation, where the degree of rotation is encoded by the text query features.

Key methodological contributions include:

  • Complex Space Projection: Mapping image and text features into a complex space allows for a more natural representation of transformations and rotations.
  • Compositional Function: A function f(z, q) learns to compose the image (z) and text (q) features, incorporating both a direct mapping to the complex space and a convolutional mapping for local feature interactions.
  • Rotational Symmetry Loss: A novel loss function is introduced that enforces symmetry by requiring the composition of the target image features with the complex conjugate of the text features to be similar to the original query image features. This encourages a more robust and consistent representation.
  • Autoencoder Framework: The model is trained within an autoencoder framework, using reconstruction losses for both image and text to act as regularizers and ensure the composed representation retains essential information.

Datasets

The proposed ComposeAE model is evaluated on three real-world datasets:

  • MIT-States: A dataset of diverse images described by an adjective (state) and a noun (category), used for tasks like changing an image from "unripe tomato" to "ripe tomato."
  • Fashion200k: A large dataset of fashion images with human-annotated captions, covering categories like dresses, tops, and skirts.
  • Fashion IQ: A challenging dataset designed to represent real-world scenarios, featuring natural language queries with an average length of 13.5 words, making it more complex than typical attribute-based queries.

The paper also introduces enhanced baselines for TIRG to ensure fair comparison, including variants using BERT for text encoding and full text captions.

Key Insights and Ideas

  • Complex Space for Composition: The hypothesis that modeling image-text composition as rotations in a complex space is more effective than simple concatenation or gating in real space is strongly supported by the results. This allows for a more principled way to encode transformations.
  • BERT's Advantage for Complex Queries: Using BERT for text feature extraction, which benefits from joint conditioning on left and right context, significantly improves performance, especially on datasets with longer and more descriptive queries like Fashion IQ. This highlights the limitations of simpler models like LSTMs for nuanced language.
  • Rotational Symmetry as Implicit Regularization: The proposed rotational symmetry loss acts as an implicit regularizer, forcing the model to learn better internal representations and improving generalization.
  • Fashion IQ as a Realistic Benchmark: The significant performance drop on Fashion IQ compared to other datasets underscores the difficulty of handling complex, natural language queries and the limitations of existing methods in such scenarios.

Key Results

ComposeAE consistently outperforms the state-of-the-art method TIRG and its enhanced variants across all three datasets.

  • Fashion200k: ComposeAE achieves a significant improvement of 30.12% over the original TIRG on the R@10 metric.
  • MIT-States: ComposeAE shows an 11.13% improvement over the original TIRG on the R@10 metric.
  • Fashion IQ: ComposeAE achieves 2.61% and 3.82% better performance than the second-best method in terms of R@10 and R@100, respectively. Notably, it outperforms TIRG with complete text query by 253% on R@10 for this dataset.
  • Ablation Studies: The rotational symmetry loss (LSYM) significantly impacts performance, especially on Fashion IQ (12.38% decrease without it). Mapping to a complex space also shows a consistent drop in performance when removed.

Limitations and Future Work

  • MIT-States Simplicity: The impact of the rotational symmetry loss on MIT-States was less significant, possibly due to the simplicity of its text queries (average 2 words). Further investigation is suggested.
  • Real Space vs. Complex Space: While the complex space mapping is beneficial, the paper notes that for fashion datasets, the performance difference between convolutional and fully-connected mappings was not as pronounced, suggesting that further exploration of these mapping strategies could be beneficial.
  • Future Work: The authors suggest that the model's ability to learn "internal" representations of text features in the complex space is promising. Further research could explore how these learned representations can be leveraged for other tasks. The paper also highlights the need for further investigation into the impact of LSYM on simpler query cases.

Summary updated

BibTeX
@inproceedings{anwaar2021compositional,
  author = {Muhammad Umer Anwaar and Egor Labintcev and Martin Kleinsteuber},
  title = {{Compositional Learning of Image-Text Query for Image Retrieval}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure