Outfit Compatibility Prediction and Diagnosis with Multi-Layered Comparison Network

Xin Wang, Bo Wu, Yueqi Zhong

TL;DR

This paper introduces a novel framework, the Multi-Layered Comparison Network (MCN), that not only predicts fashion outfit compatibility but also diagnoses why an outfit is incompatible. By leveraging multi-layered CNN features and backpropagation gradients, MCN offers interpretable insights into fashion choices, going beyond simple prediction to provide actionable explanations.

What Problem Does This Paper Address?

Predicting whether a set of fashion items will look good together (outfit compatibility) is a challenging task. While existing methods can predict compatibility, they often lack transparency. This means users and designers don't understand why an outfit is deemed compatible or incompatible, limiting the model's persuasiveness and usefulness for tasks like outfit revision or personalized recommendations. The paper aims to bridge this gap by providing both prediction and diagnosis, explaining the reasoning behind the compatibility score.

Methods

The core of the proposed approach is the Multi-Layered Comparison Network (MCN). It consists of several key components:

  • Multi-Layered Feature Extractor: This uses a Convolutional Neural Network (CNN), specifically ResNet-50, to extract features at different layers. This is crucial because lower layers capture low-level details like color and texture, while higher layers capture more semantic information like style.
  • Comparison Module at Multiple Layers: Instead of just using the final CNN layer, MCN computes pairwise similarities between items at various feature extraction layers. This allows the model to consider compatibility aspects from low-level (e.g., color clash) to high-level (e.g., style mismatch).
  • Outfit Diagnosis by Gradients: To explain the prediction, the framework uses backpropagation gradients. The gradient of the compatibility score with respect to each pairwise similarity is computed. A large gradient indicates that a particular pairwise similarity is highly influential in determining incompatibility, thus pinpointing problematic item pairings.
  • Visual Semantic Embedding (VSE): This technique is used to handle multimodal information (images and text descriptions) by learning a common representation space for fashion items.
  • Projected Embedding (PE): To address issues with standard similarity metrics (like cosine similarity) that can over-compress compatibility or violate triangle inequality, MCN employs type-specific projections. This means similarity calculations are conditioned on the types of items being compared (e.g., top vs. bottom, shoe vs. accessory).
  • MLP Predictor: A multi-layer perceptron takes the aggregated pairwise similarities from all layers and predicts the final outfit compatibility score.

Datasets

The authors introduce a new dataset called Polyvore-T. This dataset is derived from the original Polyvore dataset but is cleaned and type-labeled. It filters out irrelevant items and groups the remaining 381 categories into 5 main types (top, bottom, shoe, bag, accessory). Outfits with fewer than 3 items are also removed. This type-labeling is essential for the projected embedding mechanism.

Key Insights and Ideas

  • Diagnosis is as important as prediction: The paper argues that for fashion, understanding why something works (or doesn't) is critical for user trust and practical application.
  • Hierarchy matters in fashion: Compatibility isn't just about high-level style; low-level features like color and texture play a significant role and need to be considered at different stages of feature extraction.
  • Pairwise comparisons are fundamental: The core of outfit compatibility lies in how individual items relate to each other, not just their individual properties.
  • Gradients as explanations: Using backpropagation gradients to highlight problematic item pairs offers a computationally efficient and interpretable way to diagnose incompatibility.
  • Type-specific similarity is key: Simply comparing all items in a generic space is insufficient. Conditioning similarity calculations on item types (e.g., how a top matches a bottom) leads to more accurate and nuanced compatibility assessment.

Key Results

The MCN framework achieves state-of-the-art performance on both outfit compatibility prediction (AUC) and the more challenging fill-in-the-blank task. For example, on the Polyvore-T dataset, MCN achieved an AUC of 91.90% and a FITB accuracy of 64.35%, significantly outperforming previous methods. Ablation studies confirmed the effectiveness of each component, with the comparison module, VSE, and PE all contributing to the overall performance. Qualitative results demonstrate MCN's ability to identify problematic item pairings and suggest visually appealing revisions to outfits.

Limitations and Future Work

The authors acknowledge that while their diagnosis is interpretable, further work is needed to explore how to explain different aspects of compatibility in even better ways and to quantitatively evaluate the diagnosis results. They also suggest exploring how to automatically revise outfits based on the diagnosis, which they demonstrate with a simple substitution strategy. Future work could focus on more sophisticated revision techniques and a deeper understanding of the nuances of fashion compatibility.

Summary updated

Abstract

Existing works about fashion outfit compatibility focus on predicting the overall compatibility of a set of fashion items with their information from different modalities. However, there are few works explore how to explain the prediction, which limits the persuasiveness and effectiveness of the model. In this work, we propose an approach to not only predict but also diagnose the outfit compatibility. We introduce an end-to-end framework for this goal, which features for: (1) The overall compatibility is learned from all type-specified pairwise similarities between items, and the backpropagation gradients are used to diagnose the incompatible factors. (2) We leverage the hierarchy of CNN and compare the features at different layers to take into account the compatibilities of different aspects from the low level (such as color, texture) to the high level (such as style). To support the proposed method, we build a new type-specified outfit dataset named Polyvore-T based on Polyvore dataset. We compare our method with the prior state-of-the-art in two tasks: outfit compatibility prediction and fill-in-the-blank. Experiments show that our approach has advantages in both prediction performance and diagnosis ability.
BibTeX
@inproceedings{wang2019outfit,
  author = {Xin Wang and Bo Wu and Yueqi Zhong},
  title = {{Outfit Compatibility Prediction and Diagnosis with Multi-Layered Comparison Network}},
  booktitle = {ACM Multimedia},
  year = {2019},
  url = {https://openreview.net/forum?id=U9iKP3zhxu-},
}
Figure