Abstract
With the proliferation of e-commerce websites and the ubiquitousness of smart phones, cross-domain image retrieval using images taken by smart phones as queries to search products on e-commerce websites is emerging as a popular application. One challenge of this task is to locate the attention of both the query and database images. In particular, database images, e.g. of fashion products, on e-commerce websites are typically displayed with other accessories, and the images taken by users contain noisy background and large variations in orientation and lighting. Consequently, their attention is difficult to locate. In this paper, we exploit the rich tag information available on the e-commerce websites to locate the attention of database images. For query images, we use each candidate image in the database as the context to locate the query attention. Novel deep convolutional neural network architectures, namely TagYNet and CtxYNet, are proposed to learn the attention weights and then extract effective representations of the images. Experimental results on public datasets confirm that our approaches have significant improvement over the existing methods in terms of the retrieval accuracy and efficiency.
Introduction
The "street-to-shop" retrieval task—identifying a specific product in an e-commerce catalog from a real-world user photo—is plagued by a significant domain gap. User-generated photos are often characterized by poor lighting, cluttered backgrounds, and non-standard orientations. Conversely, professional shop images, while high-quality, often feature models wearing multiple items, making it difficult for automated systems to isolate the target product. Traditional approaches rely on expensive manual annotations like bounding boxes or landmarks. This paper proposes a method to learn attention automatically using existing metadata and cross-image context.
Methodology: The YNet Architecture
The core of the proposed solution is the YNet architecture, a deep convolutional neural network designed to bridge the domain gap by learning domain-invariant features while specializing in domain-specific attention.
The architecture introduces two primary attention mechanisms:
- TagYNet (Shop Domain): This subnetwork leverages existing product tags (e.g., "V-neck", "short sleeves") as "free" supervision. By embedding these tags into a latent space, the model learns to weight spatial locations in the shop image that correspond to the visual attributes described by the tags.
- CtxYNet (User Domain): Since user photos lack tags, the model uses a candidate shop image as context. It evaluates the feature maps of the user query against the candidate shop image representation to identify relevant regions, effectively filtering out background noise and focusing on the item of interest.
To maintain computational efficiency, the system employs a two-stage retrieval process. An initial coarse search identifies the top 256 candidates using global representations, followed by a fine-grained re-ranking using the more intensive CtxYNet attention model.
Experimental Results
The models were evaluated on the DARN and DeepFashion datasets. The results demonstrate that incorporating attention significantly boosts retrieval precision.
As shown in the precision-at-K (P@K) curves, the CtxYNIN model achieves a Top-20 Precision (P@20) of 0.242, a substantial improvement over the baseline DARN (0.112) and the landmark-reliant FashionNet (0.208). Beyond accuracy, the YNet variants are highly efficient; by aggregating spatial features into compact 512-D or 1000-D vectors, they achieve 8x to 16x better dimensionality efficiency than previous state-of-the-art methods.
Conclusion
By repurposing existing e-commerce tags and using cross-image context for attention modeling, the YNet architecture provides a scalable and accurate solution for cross-domain image retrieval. This approach eliminates the need for costly manual bounding box annotations while simultaneously improving both retrieval accuracy and system efficiency. Future work may involve refining tag selection to exclude non-visual metadata and optimizing the re-ranking stage for even lower latency.
Summary updated