Cross-Domain Image Retrieval with Attention Modeling

Xin Ji, Wei Wang, Meihui Zhang, Yang Yang

TL;DR

This paper introduces a "Y-shape" deep learning architecture designed to bridge the gap between messy user-taken photos and professional e-commerce product images. By leveraging existing product tags and using candidate shop images as "context," the model automatically learns to focus on the actual product while ignoring noisy backgrounds, significantly improving retrieval accuracy without requiring expensive manual bounding box annotations.

What Problem Does This Paper Address?

The authors tackle the "street-to-shop" retrieval problem: a user takes a photo of a piece of clothing in the real world and wants to find the exact item on an e-commerce site.

This is difficult because of the domain gap. User photos (the "user domain") are often poorly lit, taken at odd angles, and cluttered with noisy backgrounds. Shop photos (the "shop domain") are professional but often feature models wearing multiple items (e.g., a jacket, shirt, and pants), making it hard for a computer to know which specific item is the target. Previous solutions relied on human-annotated landmarks or bounding boxes to "tell" the model where to look, but these annotations are too costly to produce for massive, ever-changing e-commerce catalogs.

Methods

The researchers proposed a neural network architecture called YNet, which splits into two branches to handle domain-specific features while sharing a base for domain-invariant features. The core innovation lies in two attention modeling components:

  • TagYNet (Attention for Shop Images): Instead of manual boxes, this subnetwork uses the tags already available on shopping sites (like "V-neck" or "short sleeves"). It embeds these tags into a latent space and uses them to weight the spatial locations of the image. If a feature at a specific location matches the "V-neck" tag embedding, that area gets higher attention.
  • CtxYNet (Attention for User Images): Since user photos don't have tags, the model uses the shop image as context. It compares the feature maps of the user photo against the representation of a candidate shop image to determine which parts of the user photo are relevant.
  • Two-Stage Retrieval: To keep the system fast, they use a coarse-to-fine approach. They first perform a quick search using a simple global representation. Then, they use the computationally heavier CtxYNet to re-rank the top 256 candidates.

Datasets

The model was evaluated on two major industry benchmarks:

  1. DARN Dataset: Contains street-to-shop pairs across 20 categories with semantic attributes.
  2. DeepFashion Dataset: A massive dataset of over 800,000 images. The authors used a subset of the "street2shop" category (specifically upper clothes) to test their cross-domain performance.

Key Insights and Ideas

  • Tags as "Free" Supervision: Product tags are a goldmine of information that e-commerce sites already have. Using them as an input for attention (rather than just a target for prediction) helps the model "look" for specific physical traits.
  • Context-Dependent Querying: The "correct" way to look at a query image depends on what you are comparing it to. By making the query representation adaptive to the candidate shop image, the model can better handle deformations and occlusions in user photos.
  • Efficiency through Dimensionality: Unlike previous models (like DARN) that concatenated multiple feature layers into massive vectors, YNet aggregates spatial features into a more compact representation (e.g., 512-D or 1000-D), making it much faster for large-scale search.

Key Results

  • Accuracy: On the DARN dataset, CtxYNIN achieved a Top-20 Precision (P@20) of 0.242, significantly outperforming the original DARN model (0.112) and FashionNet (0.208).
  • DeepFashion Performance: The model maintained its lead on the DeepFashion dataset, showing a 2-4% improvement over FashionNet even without using landmark information.
  • Speed: The proposed method is roughly 8x to 16x more efficient in terms of feature dimensionality compared to baseline methods, leading to faster query times.

Limitations and Future Work

  • Noisy Metadata: The authors noted that some tags in datasets like DeepFashion are not visually relevant (e.g., "thickness of clothes"), which can confuse the attention mechanism. Filtering for purely visual tags could further improve performance.
  • Computational Cost of Re-ranking: While the two-stage approach helps, the query-specific attention (CtxYNet) still requires a second pass over candidates, which adds latency compared to a single-pass global search.
  • Scope: The study focused primarily on upper clothes; extending this to more complex categories like accessories or full-body outfits remains a future direction.

Summary updated

Abstract

With the proliferation of e-commerce websites and the ubiquitousness of smart phones, cross-domain image retrieval using images taken by smart phones as queries to search products on e-commerce websites is emerging as a popular application. One challenge of this task is to locate the attention of both the query and database images. In particular, database images, e.g. of fashion products, on e-commerce websites are typically displayed with other accessories, and the images taken by users contain noisy background and large variations in orientation and lighting. Consequently, their attention is difficult to locate. In this paper, we exploit the rich tag information available on the e-commerce websites to locate the attention of database images. For query images, we use each candidate image in the database as the context to locate the query attention. Novel deep convolutional neural network architectures, namely TagYNet and CtxYNet, are proposed to learn the attention weights and then extract effective representations of the images. Experimental results on public datasets confirm that our approaches have significant improvement over the existing methods in terms of the retrieval accuracy and efficiency.
BibTeX
@inproceedings{ji2017cross,
  author = {Xin Ji and Wei Wang and Meihui Zhang and Yang Yang},
  title = {{Cross-Domain Image Retrieval with Attention Modeling}},
  booktitle = {ACM Multimedia},
  year = {2017},
  url = {https://openreview.net/forum?id=B-nuQ5DuS8y},
}

Loading related content...

Figure