Abstract
Outfit recommendation is a complex task requiring both deep visual understanding and an intuitive grasp of aesthetic compatibility. Traditional recommendation systems often rely on sparse interaction data, failing to capture the nuanced styles and silhouettes that define fashion. This paper presents the FAshion Recommendation Machine (FARM), which treats fashion image generation as a co-supervisory task. By compelling the model to "imagine" an ideal matching item, FARM extracts richer aesthetic features, leading to state-of-the-art performance on large-scale fashion datasets like FashionVC and ExpFashion.
Introduction
The core challenge in automated fashion recommendation lies in modeling "compatibility"—a subjective human notion influenced by color, texture, style, and design. Most existing models utilize a simple recommendation loss, which indicates whether two items were previously paired. However, this signal is often too sparse to teach a neural network the intricate details of fashion design.
To bridge this gap, the authors propose that a truly effective recommender should be able to visualize what a matching item looks like. If a system can generate a plausible pair of jeans for a given shirt, it has likely captured the essential aesthetic properties required for high-quality recommendation.
The FARM Framework
The proposed FAshion Recommendation Machine (FARM) is an end-to-end framework consisting of two primary components: a Fashion Generator and a Fashion Recommender.
1. Fashion Generator
The "Imagination" module takes an input item (e.g., a top $I_t$) and a textual description of the desired match (e.g., "skinny regular jeans"). It employs a Variational Transformer to map these inputs into a latent Gaussian distribution. A Deconvolutional Neural Network (DCNN) then samples from this distribution to generate a "reference" image ($I_g$). This generative process forces the visual encoders to capture high-level aesthetic information that goes beyond simple pixel-wise similarity.
2. Fashion Recommender
The "Matching" module evaluates candidate items ($I_b$) against the input. A key innovation here is Layer-to-Layer Matching. Instead of comparing only the final feature vectors, FARM compares the internal feature maps of the candidate item's encoder with the corresponding layers of the generator. This "internal bridge" ensures that the recommender leverages the specific aesthetic knowledge (like neckline shapes or fabric washes) learned during the generation phase.
3. Co-supervision
The entire system is trained jointly. The generation loss (measuring the quality of the "imagined" match) and the recommendation loss (ranking the actual correct match) work in tandem to refine the visual encoders, creating a more robust representation of fashion compatibility.
Experimental Setup and Datasets
The effectiveness of FARM was evaluated using two large-scale datasets curated from the Polyvore fashion community.
- FashionVC: A standard benchmark containing approximately 20,000 outfits.
- ExpFashion: A significantly larger dataset with over 850,000 outfits and more detailed textual descriptions, providing a rigorous test for the model's ability to handle complex, multi-modal inputs.
Key Results and Insights
FARM demonstrated superior performance across multiple metrics, consistently outperforming strong baselines such as DVBPR and BPR-DAE.
- State-of-the-Art Accuracy: On the expansive ExpFashion dataset, FARM achieved an AUC of 88.4% and an MRR of 24.3% for bottom recommendations. This represents a significant leap over the previous best baseline (DVBPR), which achieved an 83.7% AUC.
- The Power of Imagination: Ablation studies (comparing FARM to a recommendation-only version, FARM-R) confirmed that the generation task is the primary driver of performance gains. The "imagination" signal provides the dense supervision necessary for the model to learn fine-grained style features.
- Qualitative Success: While the generated reference images are sometimes blurry, they accurately reflect the requested colors, styles, and categories. This visual "prototype" serves as an effective guide for the recommender to find the best real-world match.
Conclusion and Future Work
FARM proves that generative tasks can serve as powerful teachers for discriminative recommendation systems. By forcing a model to visualize compatibility, we can unlock deeper insights into fashion aesthetics.
Future research directions include improving generation quality by enhancing the DCNN to produce higher-fidelity reference images and expanding the outfit scope beyond top-bottom pairs to recommend full ensembles including accessories and footwear. This approach marks a significant step toward more intuitive and aesthetically-aware AI assistants in the fashion domain.
Summary updated