Lost Your Style? Navigating with Semantic-Level Approach for Text-to-Outfit Retrieval

Junkyu Jang, Eugene Hwang, Sung-Hyuk ParkWACV 2024 · OpenReview

TL;DR

This paper introduces a novel "text-to-outfit retrieval" task for fashion recommendation, aiming to generate a complete outfit based solely on a user's textual description. Their proposed CLIP4Outfit model achieves state-of-the-art performance by learning semantic relationships across three levels: item, style, and outfit, outperforming existing text-video retrieval benchmarks.

What Problem Does This Paper Address?

Traditional fashion recommendation systems often require users to have some existing knowledge of fashion or to select initial items, limiting their utility for those less fashion-savvy. Existing methods focus on predicting outfit compatibility or suggesting a single complementary item to an incomplete set. However, there's a gap in systems that can directly translate a user's natural language description of a desired look into a complete, stylish outfit. This is akin to how a human fashion stylist operates, understanding nuanced preferences and translating them into a cohesive ensemble.

Methods

The core of the proposed solution is the CLIP4Outfit framework, which adapts contrastive language-image pretraining (CLIP) principles to the fashion domain. The model operates on three progressive semantic levels:

  • Item-level Interaction: This initial stage focuses on learning cross-modal similarity between individual fashion item images and text tokens from the user's description. It uses a BERT-like model for text encoding and a Vision Transformer (ViT) for image encoding, followed by a Weighted Tokenwise Interaction (WTI) module to capture pairwise correlations.
  • Style-level Interaction: This novel level aims to capture the "style" inherent in an outfit, which is determined by the combination of items. It uses transformer encoders to process item and text embeddings, then employs a K-means clustering algorithm to extract style centroids. This is a key departure from traditional text-video models that might focus on temporal sequences.
  • Outfit-level Interaction: The final stage aggregates the style-level representations to form a global outfit representation. This is achieved using transformer encoders and mean pooling, culminating in an outfit-level similarity computation.

The overall training objective combines contrastive losses from each of these three levels, weighted by hyperparameters.

Datasets

The paper utilizes two publicly available datasets for training and evaluation:

  • Maryland Polyvore: Contains 164,379 fashion items, split into 17,316 training, 1,497 validation, and 3,076 testing outfit sets.
  • Polyvore Outfit: A larger dataset with 365,054 fashion items, divided into 53,306 training, 5,000 validation, and 10,000 testing outfit sets.

These datasets provide both images of fashion items within an outfit and the overall textual description of the outfit.

Key Insights and Ideas

  • Fashion is about "Style," not just Items: The paper highlights that an outfit's essence lies not just in individual items but in the emergent "style" created by their combination. This led to the development of the style-level interaction module.
  • Semantic Levels are Crucial: Decomposing the problem into item, style, and outfit levels allows for a more nuanced understanding of the relationship between text and fashion, mirroring how a human stylist might think.
  • Adapting CLIP for Outfits: The success of CLIP in general vision-language tasks is leveraged, but the authors demonstrate its effectiveness can be enhanced by tailoring the architecture to the unique structure of outfit data (sets of items, not temporal sequences).
  • Greedy Initialization for K-means: For the style extraction, a greedy centroid initialization method for K-means is proposed and shown to be more effective than random initialization, suggesting that the initial selection of centroids significantly impacts the quality of style representation.

Key Results

  • State-of-the-Art Performance: CLIP4Outfit significantly outperformed existing state-of-the-art models on both the Maryland Polyvore and Polyvore Outfit datasets, even when compared to models specialized for text-video retrieval.
  • Notable Improvement: On the Maryland Polyvore set, the model achieved 24.92% R@10, an improvement of 10.8% over the next best model.
  • Ablation Study Validation: All three levels (item, style, outfit) were found to contribute significantly to the model's performance, with the style-level module showing a notable improvement of up to 7.2% in R@10 and outfit-level aggregation adding another 9%.
  • User Study Success: A user study indicated high annotator satisfaction (91.17% expressed satisfaction) and a preference for the model's suggestions over ground truth in 51.93% of cases, suggesting the model's ability to capture subjective style preferences.

Limitations and Future Work

  • Subjectivity of Fashion: The authors acknowledge that fashion style is inherently subjective, which is why they conducted a user study to complement quantitative metrics.
  • Data Availability: The performance of the transformer-based architecture is expected to improve with the increased availability of high-quality text-to-outfit pairs.
  • Future Directions: The paper implicitly suggests that further exploration of how to best model the intricate relationships within and between these semantic levels could lead to even better performance. The qualitative analysis also provides a foundation for future work on explainability in fashion recommendation.

Summary updated

Abstract

Fashion stylists have historically bridged the gap between consumers’ desires and perfect outfits, which involve intricate combinations of colors, patterns, and materials. Although recent advancements in fashion recommendation systems have made strides in outfit compatibility prediction and complementary item retrieval, these systems rely heavily on pre-selected customer choices. Therefore, we introduce a groundbreaking approach to fashion recommendations: text-to-outfit retrieval task that generates a complete outfit set based solely on textual descriptions given by users. Our model is devised at three semantic levels-item, style, and outfit-where each level progressively aggregates data to form a coherent outfit recommendation based on textual input. Here, we leverage strategies similar to those in the contrastive language-image pretraining model to address the intricate-style matrix within the outfit sets. Using the Maryland Polyvore and Polyvore Outfit datasets, our approach significantly outperformed state-of-the-art models in text-video retrieval tasks, solidifying its effectiveness in the fashion recommendation domain. This research not only pioneers a new facet of fashion recommendation systems, but also introduces a method that captures the essence of individual style preferences through textual descriptions.
BibTeX
@inproceedings{jang2024lost,
  author = {Junkyu Jang and Eugene Hwang and Sung-Hyuk Park},
  title = {{Lost Your Style? Navigating with Semantic-Level Approach for Text-to-Outfit Retrieval}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=qZzCro6NDD},
}
Figure