Fashion is inherently subjective, yet we increasingly rely on AI to tell us what looks good. While deep learning models have become remarkably adept at scoring outfits, they usually act as "black boxes"—they might tell you a look is a "3/10," but they can’t tell you if it’s because the shoes are clunky or the shirt color clashes.
In this paper, researchers from Tohoku University propose a system that doesn't just grade your outfit; it points to the specific item and the specific feature (color or shape) that is dragging the score down.
TL;DR
The authors develop an "explainable" fashion recommendation system that decomposes outfit items into human-interpretable features (color and shape/texture). By calculating an Item-Feature Influence Value (IFIV) based on gradients, the system can pinpoint exactly which item or attribute makes an outfit "bad" with high accuracy.
What Problem Does This Paper Address?
Most state-of-the-art fashion AI uses Convolutional Neural Networks (CNNs) to process raw pixels. While accurate, these models offer zero transparency. If a user receives a "bad" recommendation, they don't know why, which limits the utility of the AI in real-world styling apps.
Existing attempts at "explainable" fashion often require massive amounts of manual, human-annotated data (e.g., "this shirt is 'bohemian'"). This paper seeks a middle ground: providing explanations using primitive, easily extracted features (color and shape) without needing expensive human labels for every attribute.
Methods
The authors redesigned the standard "outfit grader" architecture to prioritize interpretability:
- Interpretable Feature Extraction: Instead of feeding raw images into a black-box CNN, they extract two specific types of features:
- Color: They use K-means clustering to find the three most dominant colors in an item.
- Shape and Texture: They create "edge images" using Canny edge detection and filters to capture the silhouette and texture of the garment, which are then embedded using a pretrained CNN.
- The Outfit Grader: These features are fed into a neural network that learns to predict a "goodness score."
- Item-Feature Influence Value (IFIV): This is the core innovation. To explain a score, the system calculates the gradient of the output score with respect to each input feature. By multiplying the feature value by its gradient ($x \cdot \nabla s$), they quantify how much that specific feature "pushed" the score up or down.
- Temperature Scaling: They found that standard models are often "overconfident." They applied temperature scaling to calibrate the scores, ensuring the output represents a true probability of "fashionability."
Datasets
The researchers used the Polyvore409k dataset, a large-scale collection of outfits curated by users on the (now-defunct) Polyvore website. It contains a wide variety of items across categories like "upper," "lower," "feet," and "accessories."
Key Insights and Ideas
- The Accuracy-Interpretability Trade-off: Moving from raw pixels to hand-engineered "interpretable" features causes a slight dip in performance. However, the authors argue this ~3% drop is a fair price to pay for a system that can actually communicate with a user.
- Gradients as Explanations: The use of IFIV shows that we don't necessarily need complex "attention maps" to explain a model. Simple backpropagation to the input features can tell us which part of the input the model found most offensive.
- Calibration Matters: Without temperature scaling, the "influence" values become noisy. Calibrating the model makes the relationship between features and scores much more linear and understandable.
Key Results
- Predictive Performance: The interpretable model achieved 76.36% accuracy on the test set, compared to 79.19% for the black-box baseline.
- Flaw Detection: To test the "explanation" capability, the authors took good outfits and intentionally swapped one item (or one feature) to make them "bad."
- The system identified the "bad" item with 99.51% accuracy.
- It identified the specific "bad" shape/texture with 98.99% accuracy.
- Identifying "bad" colors was slightly harder but still impressive at 81.83% accuracy.
Limitations and Future Work
- Primitive Features: The model currently only looks at color and shape. It lacks a concept of "style" (e.g., Goth vs. Preppy) or "occasion" (e.g., Formal vs. Beachwear), which are crucial for high-level fashion advice.
- Synthetic Evaluation: The evaluation relies on "replacing" items to create bad outfits. In reality, an outfit might be bad because of the interaction between all items, rather than one single "flaw."
- Personalization: The current model provides a universal "goodness" score. Future work could integrate personal user preferences to explain why an outfit is bad for a specific person.
Summary updated