SizeNet: Weakly Supervised Learning of Visual Size and Fit in Fashion Images

Nour Karessli, Romain Guigourès, Reza ShirvanyWorkshops @ CVPR 2019 · DOI

Introduction

In fashion e-commerce, "fit" remains a primary challenge. High return rates due to sizing issues are costly for retailers and frustrating for consumers. While historical purchase and return data can inform size recommendations, these systems suffer from a "cold start" problem: new items entering a catalog have no sales history. SizeNet addresses this by learning to detect visual cues of fit issues directly from product images using a weakly supervised framework.

Methodology: The Teacher-Student Framework

The core of SizeNet is a teacher-student paradigm designed to transfer "privileged information" from historical data to a visual model.

The Statistical Teacher

Since ground-truth labels for fit are subjective and noisy, a statistical model serves as the "teacher." It analyzes historical sales and returns to identify items whose return rates significantly deviate from the category average. It generates a confidence score based on the volume of data ($n$) and the number of returns ($k$), labeling items with systemic sizing issues.

The Student Network (SizeNet)

The "student" is a deep convolutional neural network (CNN) tasked with predicting fit issues using only raw pixels from product "packshots" (images on white backgrounds). The architecture utilizes a Fashion DNA (fDNA) backbone—a ResNet-variant pre-trained on fashion data—followed by a multi-layer perceptron (MLP).

The SizeNet architecture employs a statistical teacher to guide a visual student model.

Training Strategy

To handle label noise, the authors employ curriculum learning and a confidence-weighted loss function. The model initially trains on high-confidence samples where the teacher's signal is strongest, gradually incorporating noisier data. This allows the student to learn robust visual proxies for fit issues.

Experimental Setup

The framework was evaluated on a large-scale dataset from Zalando, comprising 127,044 unique items across 12 categories and approximately 500 brands. Labels were derived from real-world return reasons, such as "item did not fit."

Results and Analysis

SizeNet demonstrates strong performance in identifying garments with inherent sizing problems, effectively bridging the gap between raw pixels and complex fit characteristics.

ROC and Precision-Recall curves comparing SizeNet against ResNet and attribute-based baselines.

Performance Benchmarks

SizeNet achieved an Area Under the Curve (AUC) of 0.74, outperforming a standard ResNet baseline (AUC 0.70). Notably, it approached the performance of an "Attributes" model (AUC 0.78), which relies on over 13,000 manually curated human tags. This suggests that the CNN successfully extracts relevant features—such as garment cut, taper, and proportions—directly from images.

Category-Specific Success

The model proved particularly effective in challenging categories like Dresses and Trousers. By providing accurate predictions for items with zero sales history, SizeNet successfully addresses the cold start problem, enabling "size advice" from the moment a product is launched.

Conclusion and Future Work

SizeNet proves that fit issues are often visually detectable. By leveraging a statistical teacher to supervise a visual student, the framework provides a scalable solution for e-commerce retailers. Future research may explore using images with human models to capture fabric drape and stretch, or moving toward more granular predictions that specify the nature of the fit issue (e.g., "tight in the shoulders").

Summary updated

BibTeX
@inproceedings{karessli2019sizenet,
  author = {Nour Karessli and Romain Guigourès and Reza Shirvany},
  title = {{SizeNet: Weakly Supervised Learning of Visual Size and Fit in Fashion Images}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00046},
}
Figure