SHIFT15M: Multiobjective Large-Scale Fashion Dataset with Distributional Shifts.

Masanari Kimura, Takuma Nakamura, Yuki Saito

Abstract

Machine learning models typically rely on the assumption that training and test data share the same distribution. In real-world applications like fashion, this assumption is frequently violated as trends evolve over time. We introduce SHIFT15M, a large-scale dataset comprising 15 million fashion images spanning a decade (2010–2020). SHIFT15M is designed to evaluate models under various distributional shifts, including covariate and target shifts. It supports multiobjective learning and provides a robust benchmark for set-to-set matching tasks.

Introduction: The Challenge of Distributional Shift

Most machine learning algorithms are built on the independent and identically distributed (i.i.d.) assumption. However, in dynamic domains such as fashion, the data distribution is in constant flux. A model trained on 2013 styles may fail to generalize to 2021 trends because the underlying features—colors, silhouettes, and combinations—have shifted. This phenomenon, known as distributional shift, poses a significant challenge for deploying reliable recommendation systems.

Existing fashion datasets often lack the scale or temporal depth required to study these shifts comprehensively. SHIFT15M addresses this gap by providing a decade of real-world data from a major fashion social networking service, enabling researchers to quantify how model performance degrades as the "temporal distance" from the training set increases.

The SHIFT15M Dataset

SHIFT15M is a massive, multiobjective dataset containing over 15 million item images and 2.5 million user-created outfits. Each entry in the dataset is structured to support complex set-to-set matching tasks, where the goal is to determine the compatibility between groups of items.

Structure of the SHIFT15M dataset, showing the JSON metadata linked to individual item images within an outfit set.

The dataset includes:

  • Scale: 15,218,721 images and 2,555,147 outfits.
  • Rich Metadata: Each item is accompanied by 4,096-dimensional VGG16 features, category IDs, prices, and "like" counts.
  • Temporal Coverage: Data from 2010 to 2020 allows for the creation of training and testing splits with varying degrees of natural shift.

Methodology: Robustness via Importance Weighting

To mitigate the impact of distributional shifts, the researchers propose a strategy based on Importance Weighted Empirical Risk Minimization (IWERM). The goal is to re-weight training samples so the model prioritizes data points that more closely resemble the target test distribution.

For set-to-set matching, the authors introduced two novel weighting strategies:

  1. max-IW: Assigns the maximum importance weight of any single item in a set to the entire set.
  2. mean-IW: Assigns the average importance weight of all items in the set.

These methods were integrated into a Set Transformer architecture, which is naturally permutation-invariant, ensuring the model treats an outfit as a cohesive whole regardless of item order.

Experimental Results

The primary benchmark task is Fill-In-The-Blank (FITB), where a model must select the correct item to complete an outfit from a set of candidates.

The Performance Cliff

Experiments confirmed a significant decay in performance as the test year moved further from the training year. A standard model trained on 2013 data achieved over 92% accuracy on contemporary data but saw a steady decline in subsequent years.

Model accuracy degradation as the temporal gap between training and testing data increases.

Mitigation Strategies

The proposed importance-weighting methods successfully mitigated this decay. For example, using the max-IW strategy improved accuracy on 2017 test data from 85.5% to 86.5%. While seemingly modest, this 1% gain is significant given the scale of the dataset and the difficulty of the task. Furthermore, in regression tasks (predicting "likes"), importance weighting significantly outperformed standard methods as the magnitude of the shift (measured via Wasserstein distance) increased.

Conclusion

SHIFT15M provides a critical resource for the machine learning community to move beyond the i.i.d. assumption. By offering a large-scale, multiobjective, and temporally diverse dataset, it enables the development of more robust models capable of handling the inevitable shifts of the real world. Future work will focus on expanding model support and exploring the calibration of these models under extreme distributional changes.

Summary updated

Abstract

Many machine learning algorithms assume that the training data and the test data follow the same distribution. However, such assumptions are often violated in real-world machine learning problems. In this paper, we propose SHIFT15M, a dataset that can be used to properly evaluate models in situations where the distribution of data changes between training and testing. The SHIFT15M dataset has several good properties: (i) Multiobjective. Each instance in the dataset has several numerical values that can be used as target variables. (ii) Large-scale. The SHIFT15M dataset consists of 15million fashion images. (iii) Coverage of types of dataset shifts. SHIFT15M contains multiple dataset shift problem settings (e.g., covariate shift or target shift). SHIFT15M also enables the performance evaluation of the model under various magnitudes of dataset shifts by switching the magnitude. In addition, we provide software to handle SHIFT15M in a very simple way: this https URL.
BibTeX
@inproceedings{kimura2023shift,
  author = {Masanari Kimura and Takuma Nakamura and Yuki Saito},
  title = {{SHIFT15M: Multiobjective Large-Scale Fashion Dataset with Distributional Shifts.}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
  url = {https://openreview.net/forum?id=BF5MDc2ezC},
}
Figure