Fashion Image Retrieval with Capsule Networks

Furkan Kinli, Baris Özcan, Furkan KiraçWorkshops @ ICCV 2019 · DOI · arXiv

Introduction

Image retrieval in the fashion domain presents unique challenges. A robust system must identify that two distinct images represent the same garment despite variations in camera angle, lighting conditions, or the way the clothing drapes on a human model. Traditional Convolutional Neural Networks (CNNs), while effective at feature extraction, often struggle with the hierarchical spatial relationships between an object and its components—a phenomenon sometimes referred to as the "Picasso Problem." For instance, a CNN might classify an image as a "shirt" simply by detecting a collar and a sleeve, regardless of their relative positions.

Addressing the Spatial Hierarchy Problem

To overcome these limitations, many state-of-the-art (SOTA) models rely on expensive, manually-labeled "side information," such as clothing landmarks (e.g., the exact coordinates of a neckline or hem). This paper investigates whether Capsule Networks can provide a more elegant solution by inherently learning the "pose" and spatial hierarchy of garments without requiring additional labels.

A typical query image showing a garment from the front.
The same garment viewed from the back, illustrating the viewpoint variance challenge.

Proposed Architectures: SCCapsNet and RCCapsNet

The authors introduce two specialized architectures designed for complex fashion imagery: SCCapsNet (Stacked Convolutional) and RCCapsNet (Residual-connected).

1. Enhanced Feature Extraction

The original Capsule Network architecture was designed for simple datasets like MNIST. To handle the high-dimensional nature of fashion images, the authors replaced the shallow entry layers with deeper blocks:

  • SC Block: Utilizes multiple convolutional layers stacked without pooling to maintain high spatial resolution.
  • RC Block: Incorporates residual (skip) connections, allowing for deeper feature learning while mitigating gradient degradation.

2. Triplet-Based Learning

The network is trained using a Triplet Loss function. During training, the model processes three images simultaneously: an anchor (the query), a positive (the same item in a different shot), and a negative (a different item). The objective is to minimize the distance in the latent embedding space between the anchor and the positive while maximizing the distance to the negative.

3. Dynamic Routing

Extracted features are organized into "Primary Capsules." Unlike standard CNNs that use max-pooling to summarize information, these capsules employ a routing-by-agreement algorithm. Lower-level capsules "vote" for the presence of higher-level entities. If multiple parts agree on the orientation and presence of a specific garment feature, the connection is strengthened, allowing the model to learn spatial relationships implicitly.

Experimental Results and Insights

The models were evaluated on the DeepFashion dataset's in-shop clothing retrieval partition, using a training set of 25,000 images and a gallery of 12,000.

Performance and Efficiency

The proposed architectures demonstrated significant improvements over the FashionNet baseline. Specifically, RCCapsNet achieved a Top-20 Recall of 84.6%, compared to 76.4% for the best FashionNet variant.

Perhaps the most significant finding is the models' parameter efficiency. While some ensemble CNN models achieve higher raw recall, they often require 10M to 134M parameters. In contrast, SCCapsNet and RCCapsNet achieve competitive results with only 2.5M and 4.5M parameters, respectively. This suggests that Capsule Networks provide a much denser and more efficient representation of visual information.

Conclusion and Future Work

Capsule Networks offer a promising path forward for fashion image retrieval by providing spatial intelligence without the need for manual landmark annotations. While the current implementation of dynamic routing is computationally more intensive than standard pooling, the dramatic reduction in parameter count highlights the architecture's potential for efficient embedding-based retrieval. Future research will focus on optimizing the routing mechanism and incorporating attention layers to further bridge the performance gap with massive CNN ensembles.

Summary updated

BibTeX
@inproceedings{kinli2019fashion,
  author = {Furkan Kinli and Baris Özcan and Furkan Kiraç},
  title = {{Fashion Image Retrieval with Capsule Networks}},
  booktitle = {Workshops @ ICCV},
  year = {2019},
  doi = {10.1109/iccvw.2019.00376},
}
Figure