Abstract
Most previous work on outfit recommendation focuses on designing visual features to enhance recommendations, often neglecting user comments which are effective for generating explanations. We propose Neural Outfit Recommendation (NOR), a framework that simultaneously provides outfit recommendations and generates abstractive comments. NOR consists of two parts: outfit matching using a CNN with a mutual attention mechanism, and comment generation using a GRU with a cross-modality attention mechanism. Jointly trained in a multi-task learning framework, NOR achieves significant improvements over state-of-the-art baselines on both existing and real-world datasets. The generated comments serve as natural language explanations for the recommendations.
Introduction
Fashion recommendation systems have traditionally focused on visual compatibility—predicting whether a specific top matches a specific bottom. While effective, these systems often operate as "black boxes," offering no explanation for their choices. This lack of transparency can hinder user trust and decision-making. Furthermore, existing research largely ignores the rich semantic information in user comments, which often contain the underlying logic of style (e.g., "the colors complement each other" or "perfect for the office").
The Neural Outfit Recommendation (NOR) framework addresses these gaps by moving beyond simple matching. It simultaneously predicts outfit compatibility and generates natural language explanations, bridging the gap between visual features and stylistic reasoning.
The Neural Outfit Recommendation (NOR) Framework
The NOR framework is a multi-task learning system designed to handle both recommendation and explanation generation through three core components:
1. Image Encoding with Mutual Attention
The system uses a two-layer Convolutional Neural Network (CNN) to extract visual features from top and bottom items. Unlike standard encoders that process items in isolation, NOR employs a mutual attention mechanism. This mechanism evaluates the correlation between local regions of the top and bottom (e.g., how a pattern on a blouse relates to the texture of a skirt), creating more nuanced latent representations for compatibility.
2. Matching Decoder
This component predicts a matching score by combining the visual features with learned collaborative filtering latent factors. These representations are fed into a multi-layer perceptron (MLP) to determine the final recommendation score.
3. Generation Decoder with Cross-Modality Attention
To provide human-like explanations, NOR uses a Gated Recurrent Unit (GRU) equipped with a cross-modality attention mechanism. This allows the text generator to "pay attention" to specific visual features of the outfit while crafting a concise, natural language comment.
Experimental Evaluation
The authors evaluated NOR using two primary datasets:
- FashionVC: A benchmark dataset of approximately 20,000 outfits, used primarily for recommendation accuracy.
- ExpFashion: A new, large-scale dataset curated from Polyvore, containing over 200,000 outfits and 1 million user comments. This dataset provided the scale necessary for training complex neural text generation models.
Key Results
- Recommendation Accuracy: On the ExpFashion dataset, NOR significantly outperformed strong baselines like BPR-DAE and DVBPR. In bottom-item recommendation, NOR achieved a Mean Average Precision (MAP) of 11.54% and an AUC of 64.75%, compared to BPR-DAE's 10.09% and 61.36%.
- Explanation Quality: The generated comments achieved impressive NLP metrics, including a BLEU score of 37.21, outperforming other generative baselines like NRT (35.61).
- Ablation Study: Removing either the mutual attention or the multi-task learning component led to a drop in performance, confirming that joint training is vital for success.
Insights and Future Directions
A significant finding of this research is that multi-task learning acts as a regularizer. Training the model to generate comments actually improves its recommendation accuracy, as the textual data provides a semantic "anchor" for visual features. Furthermore, NOR generates abstractive, human-like sentences rather than simple tag-based templates, making recommendations feel like advice from a stylist.
However, some limitations remain. The model currently has a positive bias, as most training comments are favorable. It also occasionally mentions items not present in the specific pair (e.g., shoes) due to the nature of the full-outfit training data. Future work will focus on incorporating individual user preferences for personalized recommendations and expanding the system to include more accessories like jewelry and eyewear.
Conclusion
The NOR framework represents a step forward in explainable AI for fashion. By jointly optimizing for matching and commenting, it provides both high-quality recommendations and the "why" behind them, paving the way for more transparent and trustworthy fashion AI.
Summary updated