Abstract
Determining fashion compatibility is a complex task often limited by simple item-to-item comparisons. We present the Auxiliary Template-enhanced Generative Compatibility Modeling (AT-GCM) framework, which leverages a Generative Adversarial Network (GAN) to create a visual "template" of a matching item. By combining direct item-item similarity with item-template matching, AT-GCM achieves state-of-the-art performance on large-scale fashion datasets, providing a significant boost to recommendation accuracy.
1. Introduction
In the world of online fashion, "compatibility modeling"—determining if a shirt goes well with a specific pair of trousers—is a critical challenge. Traditionally, AI models solve this by mapping items into a "latent space" and measuring the distance between them. If the points are close, the items are compatible. However, these methods have a blind spot: they don't explicitly model what an "ideal" match looks like, relying purely on item-to-item comparisons.
The authors identify three main challenges in this domain:
- Integration: Seamlessly combining image generation with compatibility scoring.
- Accuracy: Generating a "bottom" template from a "top" image that effectively guides the search.
- Modality Fusion: Effectively combining visual data (images) with textual data (descriptions and categories).
2. Methodology
The proposed AT-GCM framework is an end-to-end network consisting of two main branches working in tandem to calculate a final compatibility score.
2.1 The Item-Item Path
This branch follows the standard approach of extracting features from both items. Visual features are processed via a CNN with Global Average Pooling (GAP), while textual features (titles and categories) are processed via a TextCNN. These multi-modal features are fused to create a primary compatibility score.
2.2 The Item-Template Path
The core novelty of AT-GCM is the introduction of a Generative Adversarial Network (GAN) to create an auxiliary bridge. Given an image of a "top," the generator "sketches" a compatible "bottom" template. This branch allows the model to ask: "Does candidate item B look like the ideal partner I imagined for item A?"
The generator architecture consists of an encoder, a transformer with ResNet blocks, and a decoder. To ensure high-quality template generation, the model employs several consistency mechanisms:
- Pixel-wise L1 Loss: Forces the template to be visually similar to known positive matches.
- Cycle Consistency Loss: Borrowed from CycleGAN, this ensures that a generated bottom can be used to "reverse-engineer" the original top.
- Least Squares GAN Loss: Stabilizes training and prevents vanishing gradients common in standard GANs.
3. Experiments and Results
The model was evaluated on two real-world datasets: FashionVC (20,726 outfits) and ExpFashion (a large-scale dataset of over 850,000 outfits).
AT-GCM outperformed several state-of-the-art baselines, including Bi-LSTM and BPR-DAE. On the FashionVC dataset, it achieved an AUC of 0.8587 and an MRR (Mean Reciprocal Rank) of 0.7219, representing a 3.56% improvement in AUC and a 4.87% improvement in MRR over the best-performing baseline. Qualitative tests showed that the template-enhanced model consistently promoted "ground truth" matching items to the top of recommendation lists, whereas models without the template generator often ranked incorrect item types higher.
4. Discussion and Conclusion
A key insight from this research is the importance of shape over texture. Even when the GAN produced "fuzzy" textures, the templates were highly effective because they captured critical attributes like the length of a skirt or the cut of jeans. Furthermore, while text descriptions provide useful context, the visual modality remains dominant in conveying intuitive features like silhouette and color.
Future work will focus on incorporating textual context into the generation process. By using descriptions like "office wear" to nudge the generator, the model could produce even more semantically relevant templates, further bridging the gap between visual imagination and stylistic intent.
Summary updated