Improving Outfit Recommendation with Co-supervision of Fashion Generation

Yujie Lin, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Jun Ma, Maarten de Rijke

Abstract

Outfit recommendation is a complex task requiring both deep visual understanding and an intuitive grasp of aesthetic compatibility. Traditional recommendation systems often rely on sparse interaction data, failing to capture the nuanced styles and silhouettes that define fashion. This paper presents the FAshion Recommendation Machine (FARM), which treats fashion image generation as a co-supervisory task. By compelling the model to "imagine" an ideal matching item, FARM extracts richer aesthetic features, leading to state-of-the-art performance on large-scale fashion datasets like FashionVC and ExpFashion.

Introduction

The core challenge in automated fashion recommendation lies in modeling "compatibility"—a subjective human notion influenced by color, texture, style, and design. Most existing models utilize a simple recommendation loss, which indicates whether two items were previously paired. However, this signal is often too sparse to teach a neural network the intricate details of fashion design.

To bridge this gap, the authors propose that a truly effective recommender should be able to visualize what a matching item looks like. If a system can generate a plausible pair of jeans for a given shirt, it has likely captured the essential aesthetic properties required for high-quality recommendation.

The FARM Framework

The proposed FAshion Recommendation Machine (FARM) is an end-to-end framework consisting of two primary components: a Fashion Generator and a Fashion Recommender.

The FARM architecture combines a Fashion Generator and a Fashion Recommender through a shared latent space and layer-to-layer matching.

1. Fashion Generator

The "Imagination" module takes an input item (e.g., a top $I_t$) and a textual description of the desired match (e.g., "skinny regular jeans"). It employs a Variational Transformer to map these inputs into a latent Gaussian distribution. A Deconvolutional Neural Network (DCNN) then samples from this distribution to generate a "reference" image ($I_g$). This generative process forces the visual encoders to capture high-level aesthetic information that goes beyond simple pixel-wise similarity.

2. Fashion Recommender

The "Matching" module evaluates candidate items ($I_b$) against the input. A key innovation here is Layer-to-Layer Matching. Instead of comparing only the final feature vectors, FARM compares the internal feature maps of the candidate item's encoder with the corresponding layers of the generator. This "internal bridge" ensures that the recommender leverages the specific aesthetic knowledge (like neckline shapes or fabric washes) learned during the generation phase.

3. Co-supervision

The entire system is trained jointly. The generation loss (measuring the quality of the "imagined" match) and the recommendation loss (ranking the actual correct match) work in tandem to refine the visual encoders, creating a more robust representation of fashion compatibility.

Experimental Setup and Datasets

The effectiveness of FARM was evaluated using two large-scale datasets curated from the Polyvore fashion community.

Statistics of the FashionVC and ExpFashion datasets used for training and evaluation.
  • FashionVC: A standard benchmark containing approximately 20,000 outfits.
  • ExpFashion: A significantly larger dataset with over 850,000 outfits and more detailed textual descriptions, providing a rigorous test for the model's ability to handle complex, multi-modal inputs.

Key Results and Insights

FARM demonstrated superior performance across multiple metrics, consistently outperforming strong baselines such as DVBPR and BPR-DAE.

  • State-of-the-Art Accuracy: On the expansive ExpFashion dataset, FARM achieved an AUC of 88.4% and an MRR of 24.3% for bottom recommendations. This represents a significant leap over the previous best baseline (DVBPR), which achieved an 83.7% AUC.
  • The Power of Imagination: Ablation studies (comparing FARM to a recommendation-only version, FARM-R) confirmed that the generation task is the primary driver of performance gains. The "imagination" signal provides the dense supervision necessary for the model to learn fine-grained style features.
  • Qualitative Success: While the generated reference images are sometimes blurry, they accurately reflect the requested colors, styles, and categories. This visual "prototype" serves as an effective guide for the recommender to find the best real-world match.

Conclusion and Future Work

FARM proves that generative tasks can serve as powerful teachers for discriminative recommendation systems. By forcing a model to visualize compatibility, we can unlock deeper insights into fashion aesthetics.

Future research directions include improving generation quality by enhancing the DCNN to produce higher-fidelity reference images and expanding the outfit scope beyond top-bottom pairs to recommend full ensembles including accessories and footwear. This approach marks a significant step toward more intuitive and aesthetically-aware AI assistants in the fashion domain.

Summary updated

Abstract

The task of fashion recommendation includes two main challenges: visual understanding and visual matching. Visual understanding aims to extract effective visual features. Visual matching aims to model a human notion of compatibility to compute a match between fashion items. Most previous studies rely on recommendation loss alone to guide visual understanding and matching. Although the features captured by these methods describe basic characteristics (e.g., color, texture, shape) of the input items, they are not directly related to the visual signals of the output items (to be recommended). This is problematic because the aesthetic characteristics (e.g., style, design), based on which we can directly infer the output items, are lacking. Features are learned under the recommendation loss alone, where the supervision signal is simply whether the given two items are matched or not. To address this problem, we propose a neural co-supervision learning framework, called the FAshion Recommendation Machine (FARM). FARM improves visual understanding by incorporating the supervision of generation loss, which we hypothesize to be able to better encode aesthetic information. FARM enhances visual matching by introducing a novel layer-to-layer matching mechanism to fuse aesthetic information more effectively, and meanwhile avoiding paying too much attention to the generation quality and ignoring the recommendation performance. Extensive experiments on two publicly available datasets show that FARM outperforms state-of-the-art models on outfit recommendation, in terms of AUC and MRR. Detailed analyses of generated and recommended items demonstrate that FARM can encode better features and generate high quality images as references to improve recommendation performance.
BibTeX
@inproceedings{lin2019improving,
  author = {Yujie Lin and Pengjie Ren and Zhumin Chen and Zhaochun Ren and Jun Ma and Maarten de Rijke},
  title = {{Improving Outfit Recommendation with Co-supervision of Fashion Generation}},
  booktitle = {The Web Conference},
  year = {2019},
  doi = {10.1145/3308558.3313614},
}
Figure