Toward Explainable Fashion Recommendation

Pongsate Tangseng, Takayuki Okatani

Abstract

Many studies have been conducted so far to build systems for recommending fashion items and outfits. Although they achieve good performances in their respective tasks, most of them cannot explain their judgments to the users, which compromises their usefulness. Toward explainable fashion recommendation, this study proposes a system that is able not only to provide a goodness score for an outfit but also to explain the score by providing reason behind it. For this purpose, we propose a method for quantifying how influential each feature of each item is to the score. Using this influence value, we can identify which item and what feature make the outfit good or bad. We represent the image of each item with a combination of human-interpretable features, and thereby the identification of the most influential item-feature pair gives useful explanation of the output score. To evaluate the performance of this approach, we design an experiment that can be performed without human annotation; we replace a single item-feature pair in an outfit so that the score will decrease, and then we test if the proposed method can detect the replaced item correctly using the above influence values. The experimental results show that the proposed method can accurately detect bad items in outfits lowering their scores.

Introduction

Deep learning has significantly advanced the field of fashion recommendation, enabling models to score outfit compatibility with high accuracy. However, these models often function as "black boxes," providing a numerical score without explaining the underlying logic. For a recommendation system to be truly useful, it must provide transparency—explaining, for instance, whether an outfit is rated poorly due to a clashing color or an incompatible silhouette.

This research addresses the lack of interpretability by proposing a system that decomposes fashion items into human-interpretable features: color and shape/texture. By analyzing the influence of these specific attributes on the final recommendation score, the system can pinpoint the exact "flaw" in an outfit without requiring expensive, manually annotated datasets.

Methodology

The proposed architecture shifts away from processing raw pixels through opaque CNN layers. Instead, it utilizes a structured approach to feature extraction and scoring designed for transparency.

Interpretable Feature Extraction

Items are represented by two primary feature sets:

  1. Color: Dominant colors are extracted using K-means clustering on the item image, providing a palette representation.
  2. Shape and Texture: Edge images are generated via Canny edge detection to capture silhouettes and textures, which are then embedded using a pre-trained CNN.

Item-Feature Influence Value (IFIV)

The core of the explanatory capability lies in the IFIV. The system calculates the gradient of the outfit's "goodness score" with respect to each input feature. By calculating the product of the feature value and its corresponding gradient, the model quantifies the specific contribution of that feature to the final score.

The proposed model architecture, illustrating the flow from interpretable feature extraction to gradient-based influence calculation (IFIV).

To ensure these influence values are reliable, the authors applied temperature scaling. This calibration step corrects for the overconfidence typical in deep learning models, ensuring the output represents a true probability of "fashionability" and making the relationship between features and scores more linear.

Experimental Results

The model was trained and tested on the Polyvore409k dataset. The transition to interpretable features resulted in a minor performance trade-off: the interpretable model achieved an accuracy of 76.36%, compared to 79.19% for a non-interpretable black-box baseline. The authors argue this ~3% drop is a fair price for a system that can communicate with a user.

Flaw Detection Performance

To evaluate the system's ability to explain "bad" outfits, the researchers conducted a flaw detection experiment. They intentionally modified "good" outfits by swapping a single item or feature to lower the score and then tested if the model could identify the change using IFIV.

Visual examples of the system's ability to detect flaws at the item, shape-texture, and color levels.

The results demonstrate high precision in identifying specific issues:

  • Item-level detection: 99.51% accuracy.
  • Shape/Texture flaw detection: 98.99% accuracy.
  • Color flaw detection: 81.83% accuracy.

Conclusion

This study presents a significant step toward transparent fashion AI. By leveraging gradient-based influence values on primitive, interpretable features, the system provides meaningful explanations for its recommendations. While currently limited to color and shape, future iterations could incorporate higher-level semantic concepts and personalized user preferences to further enhance the utility of automated fashion advice.

Summary updated

Abstract

Many studies have been conducted so far to build systems for recommending fashion items and outfits. Although they achieve good performances in their respective tasks, most of them cannot explain their judgments to the users, which compromises their usefulness. Toward explainable fashion recommendation, this study proposes a system that is able not only to provide a goodness score for an outfit but also to explain the score by providing reason behind it. For this purpose, we propose a method for quantifying how influential each feature of each item is to the score. Using this influence value, we can identify which item and what feature make the outfit good or bad. We represent the image of each item with a combination of human-interpretable features, and thereby the identification of the most influential item-feature pair gives useful explanation of the output score. To evaluate the performance of this approach, we design an experiment that can be performed without human annotation; we replace a single item-feature pair in an outfit so that the score will decrease, and then we test if the proposed method can detect the replaced item correctly using the above influence values. The experimental results show that the proposed method can accurately detect bad items in outfits lowering their scores.
BibTeX
@inproceedings{tangseng2020toward,
  author = {Pongsate Tangseng and Takayuki Okatani},
  title = {{Toward Explainable Fashion Recommendation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=oleeAhKqXus},
}
Figure