SHIFT15M: Fashion-specific dataset for set-to-set matching with several distribution shifts

Masanari Kimura, Takuma Nakamura, Yuki Saito

Abstract

Set-to-set matching is the problem of matching two different sets of items based on some criteria. Especially when each item in the set is high-dimensional, such as an image, set-to-set matching is treated as one of the applied problems to be solved by utilizing neural networks. Most machine learning-based set-to-set matching generally assumes that the training and test data follow the same distribution. However, such assumptions are often violated in real-world machine learning problems. In this paper, we propose SHIFT15M, a dataset that can be used to properly evaluate set-to-set matching models in situations where the distribution of data changes between training and testing. Some benchmark experiments show that the performance of naive methods drops due to the effects of the distribution shift.

Introduction

In machine learning, the assumption of stationarity—that training and test data share the same distribution—is frequently violated in dynamic domains like fashion. Trends evolve continuously; what was considered a cohesive outfit in 2010 may be perceived as outdated by 2020. This phenomenon, known as distribution shift, poses a significant challenge for set-to-set matching tasks, such as completing a fashion outfit.

This paper introduces SHIFT15M, a large-scale dataset specifically designed to benchmark models against natural distribution shifts over a decade. By providing temporal metadata, SHIFT15M allows researchers to measure how model performance degrades as styles change and to develop robust adaptation strategies.

The SHIFT15M Dataset

SHIFT15M is a comprehensive collection of fashion data sourced from a Japanese social networking service. It contains:

  • Scale: Over 2.5 million outfits comprising approximately 15.2 million individual items.
  • Temporal Span: Data spanning from 2010 to 2020.
  • Features: Items are represented by 4096-dimensional features extracted using a pre-trained VGG16 model to ensure privacy and computational efficiency.
  • Metadata: Includes price, category hierarchies, user engagement metrics (likes), and timestamps.

Unlike static datasets like Fashion MNIST, SHIFT15M captures the gradual, human-driven evolution of culture, providing a more realistic testbed for production-grade AI systems than synthetic noise-based shifts.

Methodology: Importance Weighting for Sets

To address distribution shift, the authors employ Importance Weighted Empirical Risk Minimization (IWERM). While standard Empirical Risk Minimization (ERM) treats all training samples equally, IWERM assigns higher weights to training samples that more closely resemble the target test distribution.

The paper proposes two specific weighting strategies for sets of items:

  1. max-IW: The weight of a set is determined by the item within it that has the highest density ratio (importance).
  2. mean-IW: The weight is the average importance of all items in the set.

Importance weights are estimated using binary classifiers trained to distinguish between data from different years. If a classifier can easily distinguish between two years, it indicates a significant distribution shift.

Analysis of Distribution Shift

The authors quantified the "distance" of time by training classifiers to distinguish between pairs of years. Results indicate that the further apart two years are, the more distinct their fashion distributions become. For instance, a classifier distinguishing 2010 from 2011 achieves roughly 62% accuracy, whereas distinguishing 2010 from 2020 reaches approximately 85% accuracy.

Binary classifier accuracy showing the increase in distribution shift over time.

This confirms that fashion features undergo a gradual, continuous shift rather than abrupt changes, validating the temporal structure of the dataset.

Experimental Results

The dataset was used to evaluate several architectures (Transformers, GNNs) on a Fill-In-The-Blank (FITB) task—choosing the item that best completes an outfit.

Performance Decay

Models trained on data from a specific year show a marked decline in accuracy when tested on future years. As shown in the figure below, models generally perform best when the testing year is close to the training year, with performance dropping significantly as the temporal gap increases.

Heatmap and line graph illustrating model performance decay as the testing year deviates from the training year.

Effectiveness of Adaptation

The proposed max-IW strategy consistently improved performance in shifting environments. For example, in a 2017 test scenario, it boosted the accuracy of a Cross-Affinity model from 85.5% to 86.5%. This suggests that identifying a single "trendy" or "outdated" item can be a powerful signal for adapting the entire set's distribution.

Conclusion

SHIFT15M provides a rigorous framework for studying natural distribution shifts in high-dimensional set data. While the use of pre-extracted features limits end-to-end vision research, the scale and temporal depth of the dataset are unprecedented. Future work will focus on model calibration and expanding software support to other deep learning frameworks.

Summary updated

Abstract

Set-to-set matching is the problem of matching two different sets of items based on some criteria. Especially when each item in the set is high-dimensional, such as an image, set-to-set matching is treated as one of the applied problems to be solved by utilizing neural networks. Most machine learning-based set-to-set matching generally assumes that the training and test data follow the same distribution. However, such assumptions are often violated in real-world machine learning problems. In this paper, we propose SHIFT15M, a dataset that can be used to properly evaluate set-to-set matching models in situations where the distribution of data changes between training and testing. Some benchmark experiments show that the performance of naive methods drops due to the effects of the distribution shift. In addition, we provide software to handle SHIFT15M dataset in a very simple way: https://github.com/st-tech/zozo-shift15m. The URL for the software will appear after this manuscript is published.
BibTeX
@inproceedings{kimura2023shift,
  author = {Masanari Kimura and Takuma Nakamura and Yuki Saito},
  title = {{SHIFT15M: Fashion-specific dataset for set-to-set matching with several distribution shifts}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
  url = {https://openreview.net/forum?id=BF5MDc2ezC},
}
Figure