Sure, here's a blog-style summary of the paper "Shop The Look: Building a Large Scale Visual Shopping System at Pinterest":
TL;DR
Pinterest has built a large-scale visual shopping system called "Shop The Look" that allows users to discover and purchase products directly from images. The system combines object detection for scene decomposition with visual embeddings for candidate retrieval, achieving significant improvements in user engagement and relevance through iterative development and a focus on data quality.
What Problem Does This Paper Address?
As online content becomes increasingly visual, users are looking for more intuitive ways to find products they see in images. Traditional text-based search can be limiting when visual attributes are key to describing an item. While visual search systems exist, few detail the evolution of a large-scale system tailored specifically for shopping, addressing the unique challenges of fashion and home decor, and the critical role of data labeling in achieving high relevance and engagement.
Methods
The "Shop The Look" system comprises two main modules:
- Scene Decomposition (Object Detection): This module uses a two-stage object detection architecture (similar to Faster R-CNN) to identify prominent objects within an image and their bounding boxes. Improvements focused on:
- Single-product dataset: A new dataset was created to improve detection of individual items, crucial for many shopping queries.
- Advanced architectures: Upgrading to ResNeXt101-FasterRCNN with Feature Pyramid Networks (FPN) for better scale handling.
- Multi-scale training: Augmenting training data with various image scales to improve robustness.
- Fine-grained categories: Using more detailed product categories for better filtering.
- Candidate Retrieval (Embedding): Once an object is detected, its visual embedding is used to find visually similar products from Pinterest's catalog. Key improvements include:
- Per-category indexing: Separating product indexes by category to drastically reduce irrelevant results.
- Approximate Nearest Neighbor (ANN) search: Leveraging LSH-based tokens for efficient retrieval and enabling complex search restrictions (e.g., by gender, price).
- Multi-task embedding: Extending previous work by incorporating new tasks specifically for fashion, including:
- Large-scale pretraining: Using pre-trained weights from larger models.
- Fashion matching dataset: Training on a curated dataset of fashion images and products.
- Visual fashion attributes: Adding tasks to predict color, pattern, fabric, etc., to address specific visual mismatches.
A significant emphasis is placed on label collection and platform improvements, moving from MTurk to a hybrid approach and finally to an in-house labeling team to ensure high-quality, consistent, and objective data for both training and human evaluation.
Datasets
- Detection Training Set: 251,000 images with 727,000 bounding boxes (home decor and fashion).
- Single-Product Image Dataset: A newly created weakly supervised dataset of 133,000 images from the product corpus.
- Fashion Matching Product Dataset: A cleaned dataset of 113,000 images with 9 coarse-grained and 62,000 fine-grained category labels.
- Attribute Dataset: 70,000 image crops for fashion attributes (Color, Pattern, Fabric, etc.).
- Offline Evaluation Datasets: "Scene Images" (1,200 images) and "Single-Product Images" (1,400 images) for detection. A small matching product dataset (5,000 pairs) plus 100,000 distractors for embedding offline evaluation.
Key Insights and Ideas
- Data is King (and Queen): The paper strongly emphasizes that improvements in data quality and collection methodology (especially through an in-house labeling team) were critical drivers of success, often yielding larger gains than model architecture changes alone.
- Subjectivity is a Challenge, Not a Roadblock: While fashion and home decor can be subjective, the authors found that focusing on label consistency and addressing edge cases through better training guides and dedicated labeling teams can lead to objective and reliable results.
- Iterative Improvement is Key: The system evolved significantly through multiple iterations of detection and embedding models, each building upon previous work and addressing specific identified weaknesses.
- Infrastructure Enables Modeling: Advancements in retrieval infrastructure (like per-category indexing and ANN search) were crucial for implementing complex restrictions that significantly improved relevance, demonstrating that system design is as important as model design.
Key Results
- Cumulative Gains: Over 160% increase in end-to-end human relevance judgments and over 80% increase in engagement.
- Detection Improvements: Significant mAP gains on offline evaluations, with the "+MUL+CLN" model achieving a +102% relative mAP gain on single-product images. Online A/B tests showed increased scene closeups and product click-throughs.
- Embedding Improvements: Fashion retrieval P@1 improved significantly, with the "+Fashion dataset" version showing a +79.8% relative gain. Human relevance evaluations showed substantial increases in "Similar" and "Extremely Similar" results. Online A/B tests showed positive gains in product click-through rates.
- Labeling Platform Impact: The in-house labeling team achieved 95% label consistency and 89.5% label accuracy, a significant improvement over MTurk (47% consistency, 32.5% accuracy).
Limitations and Future Work
- Label Subjectivity: While mitigated, the inherent subjectivity in fashion and home decor remains a challenge. The paper suggests that discovering and disambiguating edge cases is an ongoing process.
- Home Decor Relevance Evaluation: The paper notes that for detection experiments, they primarily used "Similar" metrics for home decor relevance due to limitations in the available evaluation setup at the time, ideally they would target "Extremely Similar" metrics.
- Future Work: The paper implies continuous improvement through further dataset expansion, model refinement, and infrastructure enhancements, particularly in addressing nuanced visual attributes and user intent. The focus on a robust labeling platform is presented as a foundational element for future work.
Summary updated