Abstract
Machine learning models typically rely on the assumption that training and test data share the same distribution. In real-world applications like fashion, this assumption is frequently violated as trends evolve over time. We introduce SHIFT15M, a large-scale dataset comprising 15 million fashion images spanning a decade (2010–2020). SHIFT15M is designed to evaluate models under various distributional shifts, including covariate and target shifts. It supports multiobjective learning and provides a robust benchmark for set-to-set matching tasks.
Introduction: The Challenge of Distributional Shift
Most machine learning algorithms are built on the independent and identically distributed (i.i.d.) assumption. However, in dynamic domains such as fashion, the data distribution is in constant flux. A model trained on 2013 styles may fail to generalize to 2021 trends because the underlying features—colors, silhouettes, and combinations—have shifted. This phenomenon, known as distributional shift, poses a significant challenge for deploying reliable recommendation systems.
Existing fashion datasets often lack the scale or temporal depth required to study these shifts comprehensively. SHIFT15M addresses this gap by providing a decade of real-world data from a major fashion social networking service, enabling researchers to quantify how model performance degrades as the "temporal distance" from the training set increases.
The SHIFT15M Dataset
SHIFT15M is a massive, multiobjective dataset containing over 15 million item images and 2.5 million user-created outfits. Each entry in the dataset is structured to support complex set-to-set matching tasks, where the goal is to determine the compatibility between groups of items.
The dataset includes:
- Scale: 15,218,721 images and 2,555,147 outfits.
- Rich Metadata: Each item is accompanied by 4,096-dimensional VGG16 features, category IDs, prices, and "like" counts.
- Temporal Coverage: Data from 2010 to 2020 allows for the creation of training and testing splits with varying degrees of natural shift.
Methodology: Robustness via Importance Weighting
To mitigate the impact of distributional shifts, the researchers propose a strategy based on Importance Weighted Empirical Risk Minimization (IWERM). The goal is to re-weight training samples so the model prioritizes data points that more closely resemble the target test distribution.
For set-to-set matching, the authors introduced two novel weighting strategies:
- max-IW: Assigns the maximum importance weight of any single item in a set to the entire set.
- mean-IW: Assigns the average importance weight of all items in the set.
These methods were integrated into a Set Transformer architecture, which is naturally permutation-invariant, ensuring the model treats an outfit as a cohesive whole regardless of item order.
Experimental Results
The primary benchmark task is Fill-In-The-Blank (FITB), where a model must select the correct item to complete an outfit from a set of candidates.
The Performance Cliff
Experiments confirmed a significant decay in performance as the test year moved further from the training year. A standard model trained on 2013 data achieved over 92% accuracy on contemporary data but saw a steady decline in subsequent years.
Mitigation Strategies
The proposed importance-weighting methods successfully mitigated this decay. For example, using the max-IW strategy improved accuracy on 2017 test data from 85.5% to 86.5%. While seemingly modest, this 1% gain is significant given the scale of the dataset and the difficulty of the task. Furthermore, in regression tasks (predicting "likes"), importance weighting significantly outperformed standard methods as the magnitude of the shift (measured via Wasserstein distance) increased.
Conclusion
SHIFT15M provides a critical resource for the machine learning community to move beyond the i.i.d. assumption. By offering a large-scale, multiobjective, and temporally diverse dataset, it enables the development of more robust models capable of handling the inevitable shifts of the real world. Future work will focus on expanding model support and exploring the calibration of these models under extreme distributional changes.
Summary updated