Explainable Fashion Recommendation with Joint Outfit Matching and Comment Generation

Yujie Lin, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Jun Ma, Maarten de Rijke

Abstract

Most previous work on outfit recommendation focuses on designing visual features to enhance recommendations, often neglecting user comments which are effective for generating explanations. We propose Neural Outfit Recommendation (NOR), a framework that simultaneously provides outfit recommendations and generates abstractive comments. NOR consists of two parts: outfit matching using a CNN with a mutual attention mechanism, and comment generation using a GRU with a cross-modality attention mechanism. Jointly trained in a multi-task learning framework, NOR achieves significant improvements over state-of-the-art baselines on both existing and real-world datasets. The generated comments serve as natural language explanations for the recommendations.

Introduction

Fashion recommendation systems have traditionally focused on visual compatibility—predicting whether a specific top matches a specific bottom. While effective, these systems often operate as "black boxes," offering no explanation for their choices. This lack of transparency can hinder user trust and decision-making. Furthermore, existing research largely ignores the rich semantic information in user comments, which often contain the underlying logic of style (e.g., "the colors complement each other" or "perfect for the office").

The Neural Outfit Recommendation (NOR) framework addresses these gaps by moving beyond simple matching. It simultaneously predicts outfit compatibility and generates natural language explanations, bridging the gap between visual features and stylistic reasoning.

An example of a curated outfit representing a specific style, such as 'Workwear Chic', which the NOR framework aims to recommend and explain.

The Neural Outfit Recommendation (NOR) Framework

The NOR framework is a multi-task learning system designed to handle both recommendation and explanation generation through three core components:

1. Image Encoding with Mutual Attention

The system uses a two-layer Convolutional Neural Network (CNN) to extract visual features from top and bottom items. Unlike standard encoders that process items in isolation, NOR employs a mutual attention mechanism. This mechanism evaluates the correlation between local regions of the top and bottom (e.g., how a pattern on a blouse relates to the texture of a skirt), creating more nuanced latent representations for compatibility.

2. Matching Decoder

This component predicts a matching score by combining the visual features with learned collaborative filtering latent factors. These representations are fed into a multi-layer perceptron (MLP) to determine the final recommendation score.

3. Generation Decoder with Cross-Modality Attention

To provide human-like explanations, NOR uses a Gated Recurrent Unit (GRU) equipped with a cross-modality attention mechanism. This allows the text generator to "pay attention" to specific visual features of the outfit while crafting a concise, natural language comment.

Experimental Evaluation

The authors evaluated NOR using two primary datasets:

  • FashionVC: A benchmark dataset of approximately 20,000 outfits, used primarily for recommendation accuracy.
  • ExpFashion: A new, large-scale dataset curated from Polyvore, containing over 200,000 outfits and 1 million user comments. This dataset provided the scale necessary for training complex neural text generation models.

Key Results

  • Recommendation Accuracy: On the ExpFashion dataset, NOR significantly outperformed strong baselines like BPR-DAE and DVBPR. In bottom-item recommendation, NOR achieved a Mean Average Precision (MAP) of 11.54% and an AUC of 64.75%, compared to BPR-DAE's 10.09% and 61.36%.
  • Explanation Quality: The generated comments achieved impressive NLP metrics, including a BLEU score of 37.21, outperforming other generative baselines like NRT (35.61).
  • Ablation Study: Removing either the mutual attention or the multi-task learning component led to a drop in performance, confirming that joint training is vital for success.

Insights and Future Directions

A significant finding of this research is that multi-task learning acts as a regularizer. Training the model to generate comments actually improves its recommendation accuracy, as the textual data provides a semantic "anchor" for visual features. Furthermore, NOR generates abstractive, human-like sentences rather than simple tag-based templates, making recommendations feel like advice from a stylist.

However, some limitations remain. The model currently has a positive bias, as most training comments are favorable. It also occasionally mentions items not present in the specific pair (e.g., shoes) due to the nature of the full-outfit training data. Future work will focus on incorporating individual user preferences for personalized recommendations and expanding the system to include more accessories like jewelry and eyewear.

Conclusion

The NOR framework represents a step forward in explainable AI for fashion. By jointly optimizing for matching and commenting, it provides both high-quality recommendations and the "why" behind them, paving the way for more transparent and trustworthy fashion AI.

Summary updated

Abstract

Most previous work on outfit recommendation focuses on designing visual features to enhance recommendations. Existing work neglects user comments of fashion items, which have been proved to be effective in generating explanations along with better recommendation results. We propose a novel neural network framework, neural outfit recommendation (NOR), that simultaneously provides outfit recommendations and generates abstractive comments. NOR consists of two parts: outfit matching and comment generation. For outfit matching, we propose a convolutional neural network with a mutual attention mechanism to extract visual features. The visual features are then decoded into a rating score for the matching prediction. For abstractive comment generation, we propose a gated recurrent neural network with a cross-modality attention mechanism to transform visual features into a concise sentence. The two parts are jointly trained based on a multi-task learning framework in an end-to-end back-propagation paradigm. Extensive experiments conducted on an existing dataset and a collected real-world dataset show NOR achieves significant improvements over state-of-the-art baselines for outfit recommendation. Meanwhile, our generated comments achieve impressive ROUGE and BLEU scores in comparison to human-written comments. The generated comments can be regarded as explanations for the recommendation results. We release the dataset and code to facilitate future research.
BibTeX
@article{lin2020explainable,
  author = {Yujie Lin and Pengjie Ren and Zhumin Chen and Zhaochun Ren and Jun Ma and Maarten de Rijke},
  title = {{Explainable Fashion Recommendation with Joint Outfit Matching and Comment Generation}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2020},
  doi = {10.1109/tkde.2019.2906190},
}
Figure