Toward Explainable Fashion Recommendation

Pongsate Tangseng, Takayuki Okatani

Fashion is inherently subjective, yet we increasingly rely on AI to tell us what looks good. While deep learning models have become remarkably adept at scoring outfits, they usually act as "black boxes"—they might tell you a look is a "3/10," but they can’t tell you if it’s because the shoes are clunky or the shirt color clashes.

In this paper, researchers from Tohoku University propose a system that doesn't just grade your outfit; it points to the specific item and the specific feature (color or shape) that is dragging the score down.

TL;DR

The authors develop an "explainable" fashion recommendation system that decomposes outfit items into human-interpretable features (color and shape/texture). By calculating an Item-Feature Influence Value (IFIV) based on gradients, the system can pinpoint exactly which item or attribute makes an outfit "bad" with high accuracy.

What Problem Does This Paper Address?

Most state-of-the-art fashion AI uses Convolutional Neural Networks (CNNs) to process raw pixels. While accurate, these models offer zero transparency. If a user receives a "bad" recommendation, they don't know why, which limits the utility of the AI in real-world styling apps.

Existing attempts at "explainable" fashion often require massive amounts of manual, human-annotated data (e.g., "this shirt is 'bohemian'"). This paper seeks a middle ground: providing explanations using primitive, easily extracted features (color and shape) without needing expensive human labels for every attribute.

Methods

The authors redesigned the standard "outfit grader" architecture to prioritize interpretability:

  1. Interpretable Feature Extraction: Instead of feeding raw images into a black-box CNN, they extract two specific types of features:
  2. Color: They use K-means clustering to find the three most dominant colors in an item.
  3. Shape and Texture: They create "edge images" using Canny edge detection and filters to capture the silhouette and texture of the garment, which are then embedded using a pretrained CNN.
  4. The Outfit Grader: These features are fed into a neural network that learns to predict a "goodness score."
  5. Item-Feature Influence Value (IFIV): This is the core innovation. To explain a score, the system calculates the gradient of the output score with respect to each input feature. By multiplying the feature value by its gradient ($x \cdot \nabla s$), they quantify how much that specific feature "pushed" the score up or down.
  6. Temperature Scaling: They found that standard models are often "overconfident." They applied temperature scaling to calibrate the scores, ensuring the output represents a true probability of "fashionability."

Datasets

The researchers used the Polyvore409k dataset, a large-scale collection of outfits curated by users on the (now-defunct) Polyvore website. It contains a wide variety of items across categories like "upper," "lower," "feet," and "accessories."

Key Insights and Ideas

  • The Accuracy-Interpretability Trade-off: Moving from raw pixels to hand-engineered "interpretable" features causes a slight dip in performance. However, the authors argue this ~3% drop is a fair price to pay for a system that can actually communicate with a user.
  • Gradients as Explanations: The use of IFIV shows that we don't necessarily need complex "attention maps" to explain a model. Simple backpropagation to the input features can tell us which part of the input the model found most offensive.
  • Calibration Matters: Without temperature scaling, the "influence" values become noisy. Calibrating the model makes the relationship between features and scores much more linear and understandable.

Key Results

  • Predictive Performance: The interpretable model achieved 76.36% accuracy on the test set, compared to 79.19% for the black-box baseline.
  • Flaw Detection: To test the "explanation" capability, the authors took good outfits and intentionally swapped one item (or one feature) to make them "bad."
    • The system identified the "bad" item with 99.51% accuracy.
  • It identified the specific "bad" shape/texture with 98.99% accuracy.
  • Identifying "bad" colors was slightly harder but still impressive at 81.83% accuracy.

Limitations and Future Work

  • Primitive Features: The model currently only looks at color and shape. It lacks a concept of "style" (e.g., Goth vs. Preppy) or "occasion" (e.g., Formal vs. Beachwear), which are crucial for high-level fashion advice.
  • Synthetic Evaluation: The evaluation relies on "replacing" items to create bad outfits. In reality, an outfit might be bad because of the interaction between all items, rather than one single "flaw."
  • Personalization: The current model provides a universal "goodness" score. Future work could integrate personal user preferences to explain why an outfit is bad for a specific person.

Summary updated

Abstract

Many studies have been conducted so far to build systems for recommending fashion items and outfits. Although they achieve good performances in their respective tasks, most of them cannot explain their judgments to the users, which compromises their usefulness. Toward explainable fashion recommendation, this study proposes a system that is able not only to provide a goodness score for an outfit but also to explain the score by providing reason behind it. For this purpose, we propose a method for quantifying how influential each feature of each item is to the score. Using this influence value, we can identify which item and what feature make the outfit good or bad. We represent the image of each item with a combination of human-interpretable features, and thereby the identification of the most influential item-feature pair gives useful explanation of the output score. To evaluate the performance of this approach, we design an experiment that can be performed without human annotation; we replace a single item-feature pair in an outfit so that the score will decrease, and then we test if the proposed method can detect the replaced item-feature pair correctly using the above influence values. The experimental results show that the proposed method can accurately detect bad items in outfits lowering their scores.
BibTeX
@inproceedings{tangseng2020toward,
  author = {Pongsate Tangseng and Takayuki Okatani},
  title = {{Toward Explainable Fashion Recommendation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=oleeAhKqXus},
}

Loading related content...

Figure