Powering Robust Fashion Retrieval With Information Rich Feature Embeddings

Ayush Chopra, Abhishek Sinha, Hiresh Gupta, Mausoom Sarkar, Kumar Ayush, Balaji KrishnamurthyWorkshops @ CVPR 2019 · OpenReview

Abstract

Visual content-based product retrieval has become increasingly important for e-commerce. Fashion retrieval, in particular, is a challenging problem owing to a wide range of deformations of clothing items along with visual distortions in their product images. In this paper, we propose a Grid Search Network (GSN) for learning feature embeddings for fashion retrieval. The proposed approach posits the training procedure as a search problem, focused on locating matches for a reference query image in a grid containing both positive and negative images w.r.t the query. The proposed framework significantly outperforms existing state-of-art methods on benchmark fashion datasets. We also utilize a reinforcement learning based strategy to learn a specialized transformation function which further improves retrieval performance when applied over the feature embeddings. We also extend the reinforcement learning based strategy to learn custom kernel functions for SVM based classification over FashionMNIST and MNIST datasets, showing improved performance.

Introduction

Visual fashion retrieval—the task of finding a specific clothing item in a massive database based on a query image—is notoriously difficult. Unlike rigid objects, clothes are highly deformable, and product images vary wildly in lighting, pose, and occlusion. A significant challenge lies in the domain gap between professional "shop" photos and amateur "street" or "consumer" photos of the same item.

Traditional retrieval methods often rely on triplet losses, which compare a query to one positive and one negative sample at a time. This approach is often inefficient and fails to capture the complexity of a real-world search space. Furthermore, standard feature fusion techniques, such as simple vector concatenation, may not produce the most "information-rich" representations. This paper introduces the Grid Search Network (GSN) and an automated feature transformation strategy to address these limitations.

Methodology

The Grid Search Network (GSN)

The GSN architecture redefines training as a search problem. Instead of simple pairs or triplets, the network is presented with a query image and a $K \times K$ "Target Grid" containing a mix of positive and negative samples.

Overview of the Grid Search Network (GSN) architecture and training pipeline.

The network utilizes two specialized "trunks":

  1. Query Trunk: Extracts a 1024-dimensional feature vector from the query image.
  2. Target Trunk: A fully convolutional network that processes the entire $K \times K$ grid. It produces a feature volume where each spatial location corresponds to a specific image in the input grid.
An example of a 3x3 target grid used during training, with positive matches highlighted.

To train the model, the authors introduce the Grid Search Loss. This double-hinge contrastive objective concurrently compares the query vector to every image in the grid. It employs "low" and "high" threshold masks to prevent the distance between similar embeddings from collapsing to zero while maintaining a clear margin between dissimilar items. This approach mimics the actual inference task (searching a gallery), acting as a powerful form of data augmentation and forcing the model to learn more discriminative features.

RL-Based Feature Transformation

To move beyond sub-optimal human-designed feature fusion (like concatenation), the authors use an RNN controller trained via Reinforcement Learning (RL) to "discover" optimal mathematical functions for transforming and fusing embeddings. The RNN explores a library of unary operations (e.g., $sin, log, exp$) and binary operations (e.g., $max, +, \cdot$) to evolve specialized functions that maximize retrieval accuracy. For example, on the In-shop dataset, the RL agent discovered a complex function, $log(1 + exp(f_1 + f_2))$, that consistently outperformed standard methods.

Evaluation and Results

The GSN framework was evaluated on several high-profile benchmarks, demonstrating significant improvements over previous state-of-the-art (SOTA) models.

  • DeepFashion (Consumer-to-Shop): This dataset is highly challenging due to the gap between professional and amateur photos. GSN achieved a top-1 recall of 0.25, nearly doubling the performance of the previous VAM model (0.13) and tripling FashionNet (0.08).
  • DeepFashion (In-shop Retrieval): Focused on shop-to-shop matching, GSN reached a top-1 recall of 0.870, surpassing the previous best (HTL at 0.803).
  • Generalization: The RL-discovered functions proved robust beyond fashion, improving performance on the CARS196 dataset and finding custom SVM kernels for MNIST and Fashion-MNIST that outperformed standard RBF (Radial Basis Function) kernels.

Conclusion

The Grid Search Network provides a robust framework for fashion retrieval by framing training as a multi-sample search task. By combining this architecture with automated feature engineering via reinforcement learning, the authors achieved state-of-the-art results across multiple benchmarks. Future work may involve integrating adversarial hard negative mining to further enhance the discriminative power of the learned embeddings and exploring end-to-end training for the RL-based transformation search.

Summary updated

Abstract

Visual content based product retrieval has become increasingly important for e-commerce. Fashion retrieval, in particular, is a challenging problem owing to a wide range of deformations of clothing items along with visual distortions in their product images. In this paper, we propose a Grid Search Network (GSN) for learning feature embeddings for fashion retrieval. The proposed approach posits the training procedure as a search problem, focused on locating matches for a reference query image in a grid containing both positive and negative images w.r.t the query. The proposed framework significantly outperforms existing state-of-art methods on benchmark fashion datasets. We also utilize a reinforcement learning based strategy to learn a specialized transformation function which further improves retrieval performance when applied over the feature embeddings. We also extend the reinforcement learning based strategy to learn custom kernel functions for SVM based classification over FashionMNIST and MNIST datasets, showing improved performance. We highlight the generalization capabilities of this search strategy by showing performance improvement in search and attribution tasks in domains beyond fashion.
BibTeX
@inproceedings{chopra2019powering,
  author = {Ayush Chopra and Abhishek Sinha and Hiresh Gupta and Mausoom Sarkar and Kumar Ayush and Balaji Krishnamurthy},
  title = {{Powering Robust Fashion Retrieval With Information Rich Feature Embeddings}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=sXrsFEpiUq},
}
Figure