Improving Outfit Recommendation with Co-supervision of Fashion Generation

Yujie Lin, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Jun Ma, Maarten de Rijke

Abstract: The task of fashion recommendation includes two main challenges: visual understanding and visual matching. Visual understanding aims to extract effective visual features, while visual matching aims to model a human notion of compatibility. Most previous studies rely on recommendation loss alone, which often fails to capture aesthetic characteristics like style and design. To address this, we propose the FAshion Recommendation Machine (FARM), a neural co-supervision learning framework. FARM improves visual understanding by incorporating generation loss and enhances visual matching through a novel layer-to-layer matching mechanism. Extensive experiments on two datasets show that FARM outperforms state-of-the-art models in terms of AUC and MRR.

Introduction

Recommending the perfect pair of jeans to go with a specific shirt is a complex task for AI. Most existing systems treat this as a shallow "visual matching" problem, predicting a binary "match" or "no match" score based on basic characteristics like color or texture. However, a simple binary signal often misses the aesthetic nuances—the specific cut of a blazer or the texture of a skirt—that human stylists intuitively recognize.

The core idea behind the FAshion Recommendation Machine (FARM) is that a model should first be able to "imagine" what a compatible item looks like before searching for it in a catalog. By using image generation as a "co-supervision" signal, the model learns to capture fine-grained aesthetic features that traditional recommendation models miss.

Methodology

FARM is an end-to-end framework consisting of two primary components: a Fashion Generator and a Fashion Recommender.

  1. Fashion Generator: This module uses a Variational Transformer to map a "top" image and a text description of a "bottom" (e.g., "skinny regular jeans") into a latent Gaussian distribution. A Deconvolutional Neural Network (DCNN) then generates a "reference" image of the ideal matching item.
  2. Fashion Recommender: This module extracts features from candidate items in a database using a standard CNN encoder.
  3. Layer-to-Layer Matching: Instead of comparing only final embeddings, FARM compares intermediate feature maps between the generator and the recommender across multiple layers. This ensures the recommender looks for the same aesthetic characteristics the generator deemed necessary for a match.
The FARM architecture showing the interaction between the Fashion Generator and Recommender.

Experimental Setup

The model was evaluated on two large-scale datasets derived from the Polyvore fashion community:

  • FashionVC: 20,726 outfits with ~14k tops and ~13k bottoms.
  • ExpFashion: 853,991 outfits with ~168k tops and ~117k bottoms.

These datasets are unique because they include both high-quality images and natural language descriptions provided by users, allowing for multi-modal supervision.

Key Results

FARM significantly outperformed state-of-the-art baselines, including DVBPR and BPR-DAE, across all metrics. On the large-scale ExpFashion dataset, FARM achieved an AUC of 88.4% and an MRR of 24.3% for bottom recommendation, a substantial improvement over DVBPR's 83.7% AUC and 15.4% MRR.

Comparison of FARM against state-of-the-art methods on FashionVC and ExpFashion datasets.

Qualitative analysis showed that the generated "reference" images, while sometimes blurry, accurately captured the correct color, length, and category of desired items. Ablation studies further confirmed that removing the generation component (FARM-R) led to a significant drop in performance, proving that generative co-supervision is critical for high-quality recommendations.

Conclusion

FARM demonstrates that image generation can serve as a powerful "rich signal" for recommendation tasks. By forcing the model to reconstruct a visual item, the feature extractor becomes sensitive to textures, shapes, and styles that are otherwise ignored. While currently focused on top-bottom pairs, future work could expand this to full outfits including accessories, potentially requiring more complex generative architectures to handle multiple dependencies.

Summary updated

Abstract

The task of fashion recommendation includes two main challenges: visual understanding and visual matching. Visual understanding aims to extract effective visual features. Visual matching aims to model a human notion of compatibility to compute a match between fashion items. Most previous studies rely on recommendation loss alone to guide visual understanding and matching. Although the features captured by these methods describe basic characteristics (e.g., color, texture, shape) of the input items, they are not directly related to the visual signals of the output items (to be recommended). This is problematic because the aesthetic characteristics (e.g., style, design), based on which we can directly infer the output items, are lacking. Features are learned under the recommendation loss alone, where the supervision signal is simply whether the given two items are matched or not. To address this problem, we propose a neural co-supervision learning framework, called the FAshion Recommendation Machine (FARM). FARM improves visual understanding by incorporating the supervision of generation loss, which we hypothesize to be able to better encode aesthetic information. FARM enhances visual matching by introducing a novel layer-to-layer matching mechanism to fuse aesthetic information more effectively, and meanwhile avoiding paying too much attention to the generation quality and ignoring the recommendation performance. Extensive experiments on two publicly available datasets show that FARM outperforms state-of-the-art models on outfit recommendation, in terms of AUC and MRR. Detailed analyses of generated and recommended items demonstrate that FARM can encode better features and generate high quality images as references to improve recommendation performance.
BibTeX
@inproceedings{lin2019improving,
  author = {Yujie Lin and Pengjie Ren and Zhumin Chen and Zhaochun Ren and Jun Ma and Maarten de Rijke},
  title = {{Improving Outfit Recommendation with Co-supervision of Fashion Generation}},
  booktitle = {The Web Conference},
  year = {2019},
  doi = {10.1145/3308558.3313614},
}
Figure