TL;DR
This paper introduces a novel "bi-directional cross-triplet embedding" framework to tackle the challenging problem of retrieving clothing items across different domains, specifically between online shop images and real-world street images. By jointly learning to match shop items to street styles and vice-versa, the method significantly improves retrieval accuracy, addressing the inherent discrepancies in background, pose, and illumination.
What Problem Does This Paper Address?
Online shopping platforms typically present clothing items in idealized studio settings, often on professional models. This creates a disconnect for shoppers who want to know how an item will look on an ordinary person in a real-world context (e.g., on the street, in a movie). Conversely, users might see an item they like on the street and want to find it online. Existing clothing retrieval methods often struggle with this "cross-domain" challenge due to significant variations in background, lighting, and object pose between shop images and street images. This paper aims to bridge this gap by enabling seamless retrieval in both directions: shop-to-street and street-to-shop.
Methods
The core innovation is a deep bi-directional cross-triplet embedding algorithm. This approach extends the well-established triplet embedding technique, which learns to embed similar items closer together and dissimilar items further apart in a feature space. The key modifications for cross-domain retrieval are:
- Cross-Triplet Loss: The loss function is designed to handle triplets from both within the same domain (intra-domain) and across different domains (cross-domain). This explicitly encourages the model to learn similarities and differences between shop and street images.
- Weighted Loss: To emphasize the more challenging cross-domain matching, the cross-domain triplet loss is assigned a higher weight (β₂) than the intra-domain loss (β₁). The paper explores optimal ratios for this weighting.
- Bi-directional Framework: The model is trained to perform retrieval in both directions simultaneously, meaning it learns to map street images to shop images and shop images to street images.
- Fine-tuning AlexNet: Instead of training a deep network from scratch, the authors fine-tune the AlexNet architecture, leveraging its pre-trained convolutional layers and adapting its fully-connected layers for the cross-triplet embedding task. This is a practical approach given the computational cost of training deep networks and the availability of large, but not infinitely large, clothing datasets.
Datasets
The paper uses the Exact Steet2Shop dataset, which comprises:
- 20,357 street photos.
- 204,795 shop photos.
- These images cover 11 clothing categories.
Street photos are captured in uncontrolled environments, while shop photos are from online stores. The evaluation dataset is specifically constructed with items that appear in both street and shop domains.
Key Insights and Ideas
- The Discrepancy is Key: The paper highlights that learning to handle the visual discrepancies (background, pose, illumination) between domains is crucial for effective cross-domain retrieval.
- Joint Learning is Powerful: By training a single model to handle both street-to-shop and shop-to-street retrieval, the framework benefits from shared representations and a more robust understanding of clothing items across contexts.
- Weighting Matters: Assigning a higher weight to cross-domain triplets is essential for pushing the model to overcome domain shifts, rather than just relying on intra-domain similarities.
- Triplet Embedding's Adaptability: The work demonstrates that established embedding techniques like triplet embedding can be effectively adapted to new, challenging domains like cross-domain clothing retrieval with appropriate modifications.
Key Results
- Improved Accuracy: The proposed cross-triplet embedding method consistently outperforms baseline methods like AlexNet and standard triplet embedding in both street-to-shop and shop-to-street retrieval tasks across most categories.
- Competitive with State-of-the-Art: In the street-to-shop direction, the method matches or exceeds the performance of "Where [6]," a prior state-of-the-art method, even without employing complex proposal generation techniques used by "Where."
- Effective Weighting: Experiments show that a β₂/β₁ ratio between 2 and 4 yields the best performance, indicating the importance of emphasizing cross-domain learning.
Limitations and Future Work
- Dataset Size: While the dataset is substantial, the authors note it's still smaller than datasets like ImageNet, which is why fine-tuning AlexNet is preferred over training from scratch.
- Future Work: The authors plan to expand the dataset by incorporating runway fashion data (e.g., from fashion weeks) to provide users with a broader range of styles, from trend-setting to everyday wear.
Summary updated