TL;DR
This paper introduces a novel "text-to-outfit retrieval" task for fashion recommendation, aiming to generate a complete outfit based solely on a user's textual description. Their proposed CLIP4Outfit model achieves state-of-the-art performance by learning semantic relationships across three levels: item, style, and outfit, outperforming existing text-video retrieval benchmarks.
What Problem Does This Paper Address?
Traditional fashion recommendation systems often require users to have some existing knowledge of fashion or to select initial items, limiting their utility for those less fashion-savvy. Existing methods focus on predicting outfit compatibility or suggesting a single complementary item to an incomplete set. However, there's a gap in systems that can directly translate a user's natural language description of a desired look into a complete, stylish outfit. This is akin to how a human fashion stylist operates, understanding nuanced preferences and translating them into a cohesive ensemble.
Methods
The core of the proposed solution is the CLIP4Outfit framework, which adapts contrastive language-image pretraining (CLIP) principles to the fashion domain. The model operates on three progressive semantic levels:
- Item-level Interaction: This initial stage focuses on learning cross-modal similarity between individual fashion item images and text tokens from the user's description. It uses a BERT-like model for text encoding and a Vision Transformer (ViT) for image encoding, followed by a Weighted Tokenwise Interaction (WTI) module to capture pairwise correlations.
- Style-level Interaction: This novel level aims to capture the "style" inherent in an outfit, which is determined by the combination of items. It uses transformer encoders to process item and text embeddings, then employs a K-means clustering algorithm to extract style centroids. This is a key departure from traditional text-video models that might focus on temporal sequences.
- Outfit-level Interaction: The final stage aggregates the style-level representations to form a global outfit representation. This is achieved using transformer encoders and mean pooling, culminating in an outfit-level similarity computation.
The overall training objective combines contrastive losses from each of these three levels, weighted by hyperparameters.
Datasets
The paper utilizes two publicly available datasets for training and evaluation:
- Maryland Polyvore: Contains 164,379 fashion items, split into 17,316 training, 1,497 validation, and 3,076 testing outfit sets.
- Polyvore Outfit: A larger dataset with 365,054 fashion items, divided into 53,306 training, 5,000 validation, and 10,000 testing outfit sets.
These datasets provide both images of fashion items within an outfit and the overall textual description of the outfit.
Key Insights and Ideas
- Fashion is about "Style," not just Items: The paper highlights that an outfit's essence lies not just in individual items but in the emergent "style" created by their combination. This led to the development of the style-level interaction module.
- Semantic Levels are Crucial: Decomposing the problem into item, style, and outfit levels allows for a more nuanced understanding of the relationship between text and fashion, mirroring how a human stylist might think.
- Adapting CLIP for Outfits: The success of CLIP in general vision-language tasks is leveraged, but the authors demonstrate its effectiveness can be enhanced by tailoring the architecture to the unique structure of outfit data (sets of items, not temporal sequences).
- Greedy Initialization for K-means: For the style extraction, a greedy centroid initialization method for K-means is proposed and shown to be more effective than random initialization, suggesting that the initial selection of centroids significantly impacts the quality of style representation.
Key Results
- State-of-the-Art Performance: CLIP4Outfit significantly outperformed existing state-of-the-art models on both the Maryland Polyvore and Polyvore Outfit datasets, even when compared to models specialized for text-video retrieval.
- Notable Improvement: On the Maryland Polyvore set, the model achieved 24.92% R@10, an improvement of 10.8% over the next best model.
- Ablation Study Validation: All three levels (item, style, outfit) were found to contribute significantly to the model's performance, with the style-level module showing a notable improvement of up to 7.2% in R@10 and outfit-level aggregation adding another 9%.
- User Study Success: A user study indicated high annotator satisfaction (91.17% expressed satisfaction) and a preference for the model's suggestions over ground truth in 51.93% of cases, suggesting the model's ability to capture subjective style preferences.
Limitations and Future Work
- Subjectivity of Fashion: The authors acknowledge that fashion style is inherently subjective, which is why they conducted a user study to complement quantitative metrics.
- Data Availability: The performance of the transformer-based architecture is expected to improve with the increased availability of high-quality text-to-outfit pairs.
- Future Directions: The paper implicitly suggests that further exploration of how to best model the intricate relationships within and between these semantic levels could lead to even better performance. The qualitative analysis also provides a foundation for future work on explainability in fashion recommendation.
Summary updated