Explainable Noisy Label Flipping for Multi-Label Fashion Image Classification

Beatriz Quintino Ferreira, João Paulo Costeira, João Pedro GomesWorkshops @ CVPR 2021 · DOI

Abstract

SELF-ML is a novel framework designed to address the challenge of noisy or missing labels in multi-label fashion datasets. By reconstructing image features as a sparse combination of a few "cleanly" labeled basis images, the model provides an explainable mechanism to "flip" incorrect labels. This approach leverages the low-rank nature of visual feature spaces, significantly outperforming standard fine-tuning methods even when expert-verified data is extremely limited.

Introduction

In the rapidly evolving world of e-commerce, thousands of new products are uploaded daily. Manually tagging these items with multiple attributes (e.g., "V-neck," "Long Sleeved," "Formal") is both expensive and error-prone. Consequently, large-scale fashion datasets are often riddled with "noisy" labels—attributes that are either missing or incorrectly assigned.

Conceptual overview of SELF-ML: an input image is reconstructed from a basis of clean images, allowing for explainable label correction and attribute prediction.

Training Deep Convolutional Neural Networks (DCNNs) on such noisy data typically leads to poor generalization. While a small subset of data can be cleaned by experts, it is usually insufficient to train a robust model from scratch. The authors address this by proposing a method to effectively utilize massive amounts of noisy data alongside a tiny "clean" dataset to build a high-performing, explainable classifier.

Methodology

The authors propose SELF-ML, which integrates four distinct modules into a unified training pipeline:

  1. Feature Extractor: A standard CNN (VGG-16) that maps images into a high-dimensional feature space.
  2. Explainable Reconstruction Module: This core innovation assumes that the feature space of fashion images is "low-rank." The module learns a sparse vector of coefficients ($c_i$) to reconstruct an input image from a set of representative "basis" images.
  3. Noisy Label Flipping Module: Instead of blindly trusting original labels, this module analyzes the labels of the basis images used in the reconstruction. A learned flipping model decides whether to correct (flip) the original noisy labels based on visual evidence.
  4. Classifier: The final classification layer is trained using these "corrected" labels as supervision.
The SELF-ML architecture, detailing the interaction between the Feature Extractor, Explainable Reconstruction, and Noisy Label Flipping modules.

The system is trained jointly using a composite loss function ($\mathcal{L}_{total}$) that balances classification accuracy (focal loss), reconstruction quality (Huber loss), and label-flipping logic.

Experimental Evaluation

The researchers evaluated SELF-ML on a real-world e-commerce fashion dataset provided by Farfetch, consisting of approximately 60,200 images across four categories (dresses, jackets, knitwear, and tops). The dataset exhibited significant noise; expert-verified clean images had an average of 3.40 annotations, compared to only 1.28 in the noisy set. Only 3,000 images (~5%) were used as the clean reference set.

SELF-ML outperformed both a "noisy-only" baseline and standard fine-tuning approaches across all key metrics:

  • Exact Match (0-1): SELF-ML achieved 50.00%, compared to 41.73% for fine-tuning and 1.60% for the noisy-only model.
  • Jaccard Similarity: Reached 78.77%, a significant improvement over the 74.86% from fine-tuning.
  • Rare Attribute Performance: The model showed dramatic gains (5% to 20% in Average Precision) on rare and visually ambiguous labels, such as specific dress silhouettes.

Discussion and Key Insights

A primary strength of SELF-ML is its explainability. Because the model reconstructs an image using only a few basis images (typically around 40 out of 1,000), the reasoning behind a label flip is transparent. Users can inspect the basis images to understand the visual evidence influencing the model's decision.

Furthermore, the approach reinforces the idea that fashion images lie on a low-dimensional manifold, where most items are variations of prototypical styles. Unlike standard transfer learning, which often overfits on small clean sets, SELF-ML uses clean data to iteratively "fix" the noisy data throughout the training process, leading to more robust feature representations.

Conclusion

While SELF-ML demonstrates impressive results on real-world data, limitations include the use of a fixed basis size and the requirement for a small, high-quality clean set. Future work will explore dynamic basis selection and evaluation on broader public benchmarks like DeepFashion to further validate the framework's scalability.

Summary updated

BibTeX
@inproceedings{ferreira2021explainable,
  author = {Beatriz Quintino Ferreira and João Paulo Costeira and João Pedro Gomes},
  title = {{Explainable Noisy Label Flipping for Multi-Label Fashion Image Classification}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  doi = {10.1109/cvprw53098.2021.00436},
}
Figure