Abstract
Set-to-set matching is the problem of matching two different sets of items based on some criteria. Especially when each item in the set is high-dimensional, such as an image, set-to-set matching is treated as one of the applied problems to be solved by utilizing neural networks. Most machine learning-based set-to-set matching generally assumes that the training and test data follow the same distribution. However, such assumptions are often violated in real-world machine learning problems. In this paper, we propose SHIFT15M, a dataset that can be used to properly evaluate set-to-set matching models in situations where the distribution of data changes between training and testing. Some benchmark experiments show that the performance of naive methods drops due to the effects of the distribution shift.
Introduction
In machine learning, the assumption of stationarity—that training and test data share the same distribution—is frequently violated in dynamic domains like fashion. Trends evolve continuously; what was considered a cohesive outfit in 2010 may be perceived as outdated by 2020. This phenomenon, known as distribution shift, poses a significant challenge for set-to-set matching tasks, such as completing a fashion outfit.
This paper introduces SHIFT15M, a large-scale dataset specifically designed to benchmark models against natural distribution shifts over a decade. By providing temporal metadata, SHIFT15M allows researchers to measure how model performance degrades as styles change and to develop robust adaptation strategies.
The SHIFT15M Dataset
SHIFT15M is a comprehensive collection of fashion data sourced from a Japanese social networking service. It contains:
- Scale: Over 2.5 million outfits comprising approximately 15.2 million individual items.
- Temporal Span: Data spanning from 2010 to 2020.
- Features: Items are represented by 4096-dimensional features extracted using a pre-trained VGG16 model to ensure privacy and computational efficiency.
- Metadata: Includes price, category hierarchies, user engagement metrics (likes), and timestamps.
Unlike static datasets like Fashion MNIST, SHIFT15M captures the gradual, human-driven evolution of culture, providing a more realistic testbed for production-grade AI systems than synthetic noise-based shifts.
Methodology: Importance Weighting for Sets
To address distribution shift, the authors employ Importance Weighted Empirical Risk Minimization (IWERM). While standard Empirical Risk Minimization (ERM) treats all training samples equally, IWERM assigns higher weights to training samples that more closely resemble the target test distribution.
The paper proposes two specific weighting strategies for sets of items:
- max-IW: The weight of a set is determined by the item within it that has the highest density ratio (importance).
- mean-IW: The weight is the average importance of all items in the set.
Importance weights are estimated using binary classifiers trained to distinguish between data from different years. If a classifier can easily distinguish between two years, it indicates a significant distribution shift.
Analysis of Distribution Shift
The authors quantified the "distance" of time by training classifiers to distinguish between pairs of years. Results indicate that the further apart two years are, the more distinct their fashion distributions become. For instance, a classifier distinguishing 2010 from 2011 achieves roughly 62% accuracy, whereas distinguishing 2010 from 2020 reaches approximately 85% accuracy.
This confirms that fashion features undergo a gradual, continuous shift rather than abrupt changes, validating the temporal structure of the dataset.
Experimental Results
The dataset was used to evaluate several architectures (Transformers, GNNs) on a Fill-In-The-Blank (FITB) task—choosing the item that best completes an outfit.
Performance Decay
Models trained on data from a specific year show a marked decline in accuracy when tested on future years. As shown in the figure below, models generally perform best when the testing year is close to the training year, with performance dropping significantly as the temporal gap increases.
Effectiveness of Adaptation
The proposed max-IW strategy consistently improved performance in shifting environments. For example, in a 2017 test scenario, it boosted the accuracy of a Cross-Affinity model from 85.5% to 86.5%. This suggests that identifying a single "trendy" or "outdated" item can be a powerful signal for adapting the entire set's distribution.
Conclusion
SHIFT15M provides a rigorous framework for studying natural distribution shifts in high-dimensional set data. While the use of pre-extracted features limits end-to-end vision research, the scale and temporal depth of the dataset are unprecedented. Future work will focus on model calibration and expanding software support to other deep learning frameworks.
Summary updated