Abstract: The task of fashion recommendation includes two main challenges: visual understanding and visual matching. Visual understanding aims to extract effective visual features, while visual matching aims to model a human notion of compatibility. Most previous studies rely on recommendation loss alone, which often fails to capture aesthetic characteristics like style and design. To address this, we propose the FAshion Recommendation Machine (FARM), a neural co-supervision learning framework. FARM improves visual understanding by incorporating generation loss and enhances visual matching through a novel layer-to-layer matching mechanism. Extensive experiments on two datasets show that FARM outperforms state-of-the-art models in terms of AUC and MRR.
Introduction
Recommending the perfect pair of jeans to go with a specific shirt is a complex task for AI. Most existing systems treat this as a shallow "visual matching" problem, predicting a binary "match" or "no match" score based on basic characteristics like color or texture. However, a simple binary signal often misses the aesthetic nuances—the specific cut of a blazer or the texture of a skirt—that human stylists intuitively recognize.
The core idea behind the FAshion Recommendation Machine (FARM) is that a model should first be able to "imagine" what a compatible item looks like before searching for it in a catalog. By using image generation as a "co-supervision" signal, the model learns to capture fine-grained aesthetic features that traditional recommendation models miss.
Methodology
FARM is an end-to-end framework consisting of two primary components: a Fashion Generator and a Fashion Recommender.
- Fashion Generator: This module uses a Variational Transformer to map a "top" image and a text description of a "bottom" (e.g., "skinny regular jeans") into a latent Gaussian distribution. A Deconvolutional Neural Network (DCNN) then generates a "reference" image of the ideal matching item.
- Fashion Recommender: This module extracts features from candidate items in a database using a standard CNN encoder.
- Layer-to-Layer Matching: Instead of comparing only final embeddings, FARM compares intermediate feature maps between the generator and the recommender across multiple layers. This ensures the recommender looks for the same aesthetic characteristics the generator deemed necessary for a match.
Experimental Setup
The model was evaluated on two large-scale datasets derived from the Polyvore fashion community:
- FashionVC: 20,726 outfits with ~14k tops and ~13k bottoms.
- ExpFashion: 853,991 outfits with ~168k tops and ~117k bottoms.
These datasets are unique because they include both high-quality images and natural language descriptions provided by users, allowing for multi-modal supervision.
Key Results
FARM significantly outperformed state-of-the-art baselines, including DVBPR and BPR-DAE, across all metrics. On the large-scale ExpFashion dataset, FARM achieved an AUC of 88.4% and an MRR of 24.3% for bottom recommendation, a substantial improvement over DVBPR's 83.7% AUC and 15.4% MRR.
Qualitative analysis showed that the generated "reference" images, while sometimes blurry, accurately captured the correct color, length, and category of desired items. Ablation studies further confirmed that removing the generation component (FARM-R) led to a significant drop in performance, proving that generative co-supervision is critical for high-quality recommendations.
Conclusion
FARM demonstrates that image generation can serve as a powerful "rich signal" for recommendation tasks. By forcing the model to reconstruct a visual item, the feature extractor becomes sensitive to textures, shapes, and styles that are otherwise ignored. While currently focused on top-bottom pairs, future work could expand this to full outfits including accessories, potentially requiring more complex generative architectures to handle multiple dependencies.
Summary updated