SAT: Self-adaptive training for fashion compatibility prediction

Ling Xiao, Toshihiko Yamasaki

Introduction

Predicting fashion compatibility—determining if a set of clothing items "goes together"—is a complex challenge for artificial intelligence. While deep learning models excel at identifying low-level visual similarities like color or texture, they often struggle with the high-level nuances of style, silhouette, and formalness. A common failure mode occurs with "hard" samples: items that look visually similar (e.g., two black garments) but are stylistically incompatible due to conflicting aesthetics.

Researchers from the University of Tokyo address this gap with Self-Adaptive Training (SAT). Their approach shifts the focus of neural networks from "easy" matches to these challenging stylistic nuances, achieving state-of-the-art results without the need for complex attention mechanisms or auxiliary text metadata.

Methodology: Learning from Difficulty

The SAT framework is built upon a VGG13 backbone and introduces three key innovations to refine how models learn compatibility:

  1. Type-Specific Embeddings: Rather than a single global space, the model maps items into specific embedding spaces based on their category (e.g., shoes, dresses, tops). This allows the network to learn distinct compatibility rules for different types of pairings.
  2. Difficulty Score (DS): During training, the model calculates a "Difficulty Score" for each outfit. If the current model state struggles to distinguish a compatible item from an incompatible one—specifically when a "negative" item is closer in the embedding space than the "positive" one—the sample is assigned a high DS.
  3. Self-Adaptive Triplet Loss (SATL): This is the core contribution. The standard triplet loss is weighted by a matrix derived from the DS. This mechanism forces the CNN to prioritize learning from samples where it is currently failing, effectively acting as a teacher that highlights subtle style nuances.
Example of the Fill-in-the-Blank (FITB) task with distance scores.

Experimental Evaluation

The model was rigorously tested on two industry-standard benchmarks: Polyvore Outfits and Polyvore Outfits-D (a disjoint version designed to prevent the model from memorizing specific item "families").

The SAT model demonstrated superior performance in two primary tasks:

  • Fill-in-the-Blank (FITB): Selecting the correct item to complete a partial outfit.
  • Compatibility Prediction: Determining the overall aesthetic harmony of a full outfit.

On the Polyvore Outfits dataset, SAT achieved a 62.2% FITB accuracy and a 0.92 Compatibility AUC. These results outperformed more complex architectures, including those utilizing Graph Neural Networks (GNNs) or text-based metadata.

Key Insights

The success of SAT underscores a vital principle in machine learning for fashion: the "easy" problems, such as color matching, are largely solved. The frontier lies in "hard" samples where visual similarity does not imply compatibility. By focusing training on these edge cases through the SATL, the model learns to prioritize stylistic harmony over simple visual resemblance.

Furthermore, the SATL component is highly extensible. When applied to other existing architectures like CSA-Net, it consistently provided a performance boost, suggesting that adaptive weighting is a robust strategy for improving fashion recommendation systems.

Conclusion and Future Directions

While SAT proves the power of purely visual self-adaptive learning, there are clear paths for future enhancement. Integrating text descriptions could provide semantic context that images alone might miss. Additionally, moving beyond random negative sampling to more sophisticated "hard negative mining" could further refine the difficulty scoring process. Finally, accounting for temporal shifts in fashion trends remains a significant challenge for future iterations of compatibility models.

Summary updated

Abstract

This paper presents a self-adaptive training (SAT) model for fashion compatibility prediction. It focuses on the learning of some hard items, such as those that share similar color, texture, and pattern features but are considered incompatible due to the aesthetics or temporal shifts. Specifically, we first design a method to define hard outfits and a difficulty score (DS) is defined and assigned to each outfit based on the difficulty in recommending an item for it. Then, we propose a self-adaptive triplet loss (SATL), where the DS of the outfit is considered. Finally, we propose a very simple conditional similarity network combining the proposed SATL to achieve the learning of hard items in the fashion compatibility prediction. Experiments on the publicly available Polyvore Outfits and Polyvore Outfits-D datasets demonstrate our SAT’s effectiveness in fashion compatibility prediction. Besides, our SATL can be easily extended to other conditional similarity networks to improve their performance.
BibTeX
@inproceedings{xiao2022sat,
  author = {Ling Xiao and Toshihiko Yamasaki},
  title = {{SAT: Self-adaptive training for fashion compatibility prediction}},
  booktitle = {ICIP},
  year = {2022},
  doi = {10.1109/icip46576.2022.9897313},
}
Figure