Mining Fashion Outfit Composition Using an End-to-End Deep Learning Approach on Set Data

Yuncheng Li, Liangliang Cao, Jiang Zhu, Jiebo Luo

TL;DR

This paper introduces an end-to-end deep learning framework that automatically composes fashion outfits by learning to score the compatibility of item sets. By combining visual features with textual metadata and leveraging large-scale social media "popularity" data as a supervisory signal, the system achieves an 85% AUC in outfit scoring and 77% accuracy in completing outfits.

What Problem Does This Paper Address?

Fashion outfit composition is a complex, subjective task that requires balancing colors, styles, and "dressing codes." While AI has made strides in individual object recognition and retrieval (e.g., "find a similar shirt"), automatically building a coherent set of items—like a dress, bag, and shoes—remains a significant challenge.

The authors identify two main hurdles:

  1. Visual Complexity: Clothing has countless attributes (patterns, textures, cuts) that are difficult to label exhaustively.
  2. Contextual Nuance: A "good" outfit depends on subtle interactions between items. For instance, a fancy dress might not normally pair with a casual backpack, but the right pair of shoes can bridge that stylistic gap to create a trendy look.
An example of how adding a third item (shoes) can complete a non-trivial outfit composition.

Methods

The authors propose a two-part system: a Scoring Model to evaluate outfit quality and a Composition Algorithm to build new outfits.

The Scoring Model

This is a multi-modal, multi-instance deep learning architecture:

  • Multi-modal Encoders: For every item, the model extracts features from three sources: images (using a variant of AlexNet), titles (using GloVe word embeddings), and categories (using learned categorical embeddings).
  • Fusion & Pooling: Features from these modalities are fused into a single item representation. Because an outfit is an orderless set of variable size, the model uses Multi-instance Pooling (specifically mean pooling) to aggregate individual item features into a holistic "outfit embedding."
  • Classification: A final layer predicts a quality score based on this outfit embedding.
The proposed end-to-end fashion outfit scoring model architecture.

The Composition Algorithm

To create an outfit, the system uses a greedy iterative approach. Starting with a "seed" item (e.g., a skirt chosen by a user), it searches a candidate database and repeatedly adds the item that maximizes the overall outfit score until the target outfit length is reached.

Datasets

The authors introduced a massive new dataset crawled from Polyvore.com:

  • Scale: 195,262 outfits and 368,251 unique fashion items.
  • Metadata: Includes images, titles, and categories for every item.
  • Labels: Instead of expensive expert labeling, they used "likes" as a proxy for quality. They defined "popular" outfits (top 10% of likes) as positive examples and "unpopular" ones (bottom 40%) as negative examples for training.

Key Insights and Ideas

  • Text is a Powerful Signal: Surprisingly, item titles (text) were more predictive of outfit quality than images alone. The authors suggest this is because text often captures high-level semantic information (like "cocktail" or "vintage") that current computer vision models might miss.
  • Sets > Individuals: The model performs significantly better when it learns from the interaction of items during training (multi-instance learning) rather than just averaging the scores of individual items (late fusion).
  • Outfits are Sets, Not Sequences: When testing different pooling methods, simple mean pooling outperformed Recurrent Neural Networks (RNNs). This confirms that fashion outfits are best modeled as orderless sets rather than ordered sequences.
  • Multi-modality is Essential: Combining image, title, and category data consistently outperformed any single modality, proving that visual and contextual cues are complementary in fashion.

Key Results

  • Scoring Accuracy: The full multi-modal model achieved an AUC of 85.2% on the scoring task, significantly outperforming baselines like Siamese networks and traditional metric learning (LEML).
  • Composition Performance: In a "constrained composition" test—where the model had to pick the one "correct" item out of five to complete an outfit—it achieved 77% accuracy when compared against a "gold standard" set of labels provided by fashion experts.
  • Robustness: The model's performance was stable across various embedding dimensions, peaking at a relatively compact dimension of $d=8$.
Distribution and frequency of various fashion categories in the Polyvore dataset.

Limitations and Future Work

The authors acknowledge that "popularity" is a noisy proxy for fashion sense, as social media likes can be influenced by a user's follower count or post timing rather than just aesthetic merit.

Future directions include:

  • Developing more sophisticated ways to model the complex, non-linear interactions between different fashion modalities.
  • Building online systems to collect real-time user feedback on auto-generated outfits.
  • Expanding the dataset to include even more diverse styles and niche fashion communities.

Summary updated

Abstract

Composing fashion outfits involves deep under-standing of fashion standards while incorporating creativity for choosing multiple fashion items (e.g., jewelry, bag, pants, dress). In fashion websites, popular or high-quality fashion outfits are usually designed by fashion experts and followed by large audiences. In this paper, we propose a machine learning system to compose fashion outfits automatically. The core of the proposed automatic composition system is to score fashion outfit candidates based on the appearances and metadata. We propose to leverage outfit popularity on fashion-oriented websites to supervise the scoring component. The scoring component is a multimodal multiinstance deep learning system that evaluates instance aesthetics and set compatibility simultaneously. In order to train and evaluate the proposed composition system, we have collected a large-scale fashion outfit dataset with 195K outfits and 368K fashion items from Polyvore. Although the fashion outfit scoring and composition is rather challenging, we have achieved an AUC of 85% for the scoring component, and an accuracy of 77% for a constrained composition task.
BibTeX
@article{li2017mining,
  author = {Yuncheng Li and Liangliang Cao and Jiang Zhu and Jiebo Luo},
  title = {{Mining Fashion Outfit Composition Using an End-to-End Deep Learning Approach on Set Data}},
  journal = {IEEE Trans. Multimedia},
  year = {2017},
  url = {https://openreview.net/forum?id=cHmXaJgiNH5},
}

Loading related content...

Figure