Towards a Unified Framework for Visual Compatibility Prediction

Anirudh Singhal, Ayush Chopra, Kumar Ayush, Utkarsh Patel, Balaji KrishnamurthyWACV 2020 · OpenReview

Abstract

Visual compatibility prediction refers to the task of determining if a set of items go well together. Existing techniques for compatibility prediction prioritize sensitivity to type or context in item representations and evaluate using a fill-in-the-blank (FITB) task. We scale the FITB task to stresstest existing methods which highlights the need for a compatibility prediction framework that is sensitive to multiple modalities of item relationships. In this work, we introduce a unified framework for compatibility learning that is jointly conditioned on the type, context, and style. The framework is composed of TC-GAE, a graph-based network that models type & context; SAE, an autoencoder that models style; and a reinforcement-learning based search technique that incorporates these modalities to learn a unified compatibility measure. We conduct experiments on two standard datasets and significantly outperform existing state-of-the- art methods. We also present qualitative analysis and discussions to study the impact of components of the proposed framework.

Introduction

Predicting fashion compatibility is a notoriously difficult task because it involves a subjective interplay of human creativity, expertise, and aesthetic "vibe." Existing AI models typically focus on one of two signals: Type Conditioning (e.g., does a specific shirt go with these pants?) or Context (what items are frequently seen together in a dataset?).

Furthermore, the standard evaluation metric—the Fill-In-The-Blank (FITB) task—is often too simplistic. In a standard FITB, a model chooses the correct missing item from just four options. In real-world recommender systems, models must navigate thousands of choices. This paper addresses these limitations by introducing a unified framework and a more rigorous "Scaled FITB" evaluation to stress-test model robustness.

Methodology

The authors propose a three-pronged architecture designed to capture the multi-modal nature of fashion outfits:

TC-GAE: Type Conditioned Graph Autoencoder

This component treats an outfit as a graph where items are nodes. It uses a "type-co-occurrence weighted adjacency matrix" to learn representations. By recognizing that certain categories (like shoes and socks) have stronger inherent relationships than others, the model uses the surrounding context of the outfit to refine each item's embedding within specific compatibility spaces.

The TC-GAE architecture models outfits as graphs, learning embeddings within specific type-conditioned compatibility spaces.

SAE: Attentive Style Autoencoder

Beyond category and context, the SAE models the "vibe." It uses an attention mechanism to determine which items contribute most to an outfit's overall style (e.g., a blazer defining a "formal" style more than a plain tee). It learns to represent the entire outfit as a mixture of latent style "bases," allowing the model to handle subjectivity without needing manual labels like "Boho" or "Goth."

Reinforcement Learning (RL) for Fusion

Instead of manually weighting the scores from the TC-GAE and SAE, the authors use an RNN controller (an RL agent) to search for the optimal mathematical function to combine these modalities. The agent explores various operations to find the formula that yields the highest accuracy, discovering non-intuitive heuristics that outperform simple averages.

Experimental Setup and Datasets

The framework was evaluated on two gold-standard fashion datasets: Maryland Polyvore (~21,000 outfits) and UIUC Polyvore (~32,000 outfits). Crucially, the authors introduced the Scaled FITB task, where they linearly increased the number of candidate choices from 4 up to 10 to observe how model performance decays as the task becomes more difficult.

Key Results and Insights

The "Robustness Gap"

A striking insight from the study is how quickly standard models fail when the number of "distractor" items increases. While a model might achieve 90% accuracy with 4 choices, performance often drops to 40% with 10. The proposed unified approach shows a significantly slower rate of decay, suggesting it has learned more fundamental rules of compatibility.

Accuracy decay on the Scaled FITB task as the number of candidate options increases.

Performance Benchmarks

  • State-of-the-Art: The TC-GAE consistently outperformed previous benchmarks. On the UIUC-D dataset with 10 candidates, the proposed model achieved 55.45% accuracy, a significant jump over the previous best of ~49.75%.
  • The Power of Weighting: Using a weighted adjacency matrix (incorporating category co-occurrence) provided a consistent 2-3% boost, proving that knowing "what" an item is helps the model understand "where" it fits.
  • Style Matters: Adding the SAE component to the TC-GAE provided an additional performance boost, confirming that style is a distinct and necessary modality for compatibility.

Conclusion

This research highlights the necessity of a multi-modal approach to fashion AI. By jointly modeling type, context, and style, the proposed framework achieves superior robustness and accuracy. Future work may involve incorporating rich text descriptions and exploring how these graph-based methods scale to massive industrial catalogs containing millions of items.

Summary updated

Abstract

Visual compatibility prediction refers to the task of determining if a set of items go well together. Existing techniques for compatibility prediction prioritize sensitivity to type or context in item representations and evaluate using a fill-in-the-blank (FITB) task. We scale the FITB task to stresstest existing methods which highlights the need for a compatibility prediction framework that is sensitive to multiple modalities of item relationships. In this work, we introduce a unified framework for compatibility learning that is jointly conditioned on the type, context, and style. The framework is composed of TC-GAE, a graph-based network that models type & context; SAE, an autoencoder that models style; and a reinforcement-learning based search technique that incorporates these modalities to learn a unified compatibility measure. We conduct experiments on two standard datasets and significantly outperform existing state-of-the- art methods. We also present qualitative analysis and discussions to study the impact of components of the proposed framework.
BibTeX
@inproceedings{singhal2020towards,
  author = {Anirudh Singhal and Ayush Chopra and Kumar Ayush and Utkarsh Patel and Balaji Krishnamurthy},
  title = {{Towards a Unified Framework for Visual Compatibility Prediction}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=kPjhmXPI1Kt},
}

Loading related content...

Figure