Abstract
Composing fashion outfits involves deep understanding of fashion standards while incorporating creativity for choosing multiple fashion items (e.g., Jewelry, Bag, Pants, Dress). In fashion websites, popular or high-quality fashion outfits are usually designed by fashion experts and followed by large audiences. In this paper, we propose a machine learning system to compose fashion outfits automatically. The core of the proposed automatic composition system is to score fashion outfit candidates based on the appearances and meta-data. We propose to leverage outfit popularity on fashion oriented websites to supervise the scoring component. The scoring component is a multi-modal multi-instance deep learning system that evaluates instance aesthetics and set compatibility simultaneously. In order to train and evaluate the proposed composition system, we have collected a large scale fashion outfit dataset with 195K outfits and 368K fashion items from Polyvore. Although the fashion outfit scoring and composition is rather challenging, we have achieved an AUC of 85% for the scoring component, and an accuracy of 77% for a constrained composition task.
Introduction
Fashion outfit composition is a complex, subjective task that requires balancing colors, styles, and "dressing codes." While AI has made significant strides in individual object recognition and retrieval, automatically building a coherent set of items—such as a dress, bag, and shoes—remains a significant challenge. The authors identify two main hurdles: visual complexity, where clothing has countless attributes that are difficult to label exhaustively, and contextual nuance, where a "good" outfit depends on subtle interactions between items.
Methodology
The proposed system consists of two primary components: a Scoring Model to evaluate outfit quality and a Composition Algorithm to generate new outfits.
The Scoring Model
The scoring component is a multi-modal, multi-instance deep learning architecture designed to evaluate instance aesthetics and set compatibility simultaneously.
- Multi-modal Encoders: For every item, the model extracts features from three sources: images (using a variant of AlexNet), titles (using GloVe word embeddings), and categories (using learned categorical embeddings).
- Fusion & Pooling: Features from these modalities are fused into a single item representation. Because an outfit is an orderless set of variable size, the model uses Multi-instance Pooling (specifically mean pooling) to aggregate individual item features into a holistic "outfit embedding."
- Classification: A final layer predicts a quality score based on this outfit embedding.
The Composition Algorithm
To create an outfit, the system uses a greedy iterative approach. Starting with a "seed" item (e.g., a skirt chosen by a user), it searches a candidate database and repeatedly adds the item that maximizes the overall outfit score until the target outfit length is reached.
Dataset and Experimental Setup
The authors introduced a massive new dataset crawled from Polyvore.com, containing 195,262 outfits and 368,251 unique fashion items. Each item includes images, titles, and categories. Instead of expensive expert labeling, the researchers used social media "likes" as a proxy for quality. "Popular" outfits (top 10% of likes) were defined as positive examples, while "unpopular" ones (bottom 40%) served as negative examples for training.
Key Results and Insights
The experimental results yielded several critical insights into automated fashion composition:
- Textual Signals: Item titles were found to be more predictive of outfit quality than images alone, likely because text captures high-level semantic information (e.g., "cocktail" or "vintage") that visual models might miss.
- Set Modeling: The model performs significantly better when learning from item interactions during training (multi-instance learning) rather than simple late fusion of individual scores.
- Orderless Nature: Mean pooling outperformed Recurrent Neural Networks (RNNs), confirming that fashion outfits are best modeled as orderless sets rather than sequences.
- Performance: The full multi-modal model achieved an AUC of 85.2% on the scoring task. In a "constrained composition" test—where the model had to pick the one "correct" item out of five to complete an outfit—it achieved 77% accuracy.
Conclusion
This work demonstrates that an end-to-end deep learning framework can effectively capture the nuances of fashion compatibility by leveraging multi-modal data and large-scale social signals. While "popularity" is a noisy proxy for aesthetic merit, the system's high performance suggests it has learned meaningful stylistic patterns. Future work will focus on modeling non-linear interactions between modalities and incorporating real-time user feedback.
Summary updated