Winning Ticket in Noisy Image Classification.

Taehyeon Kim, Jongwoo Ko, Jinhwan Choi, Sangwook Cho, Se-Young YunarXiv 2021 · arXiv

Abstract

Modern deep neural networks (DNNs) become frail when the datasets contain noisy (incorrect) class labels. Many robust techniques have emerged via loss adjustment, robust loss function, and clean sample selection to mitigate this issue using the whole dataset. Here, we empirically observe that the dataset which contains only clean instances in original noisy datasets leads to better optima than the original dataset even with fewer data. Based on these results, we state the winning ticket hypothesis: regardless of robust methods, any DNNs reach the best performance when trained on the dataset possessing only clean samples from the original (winning ticket). We propose two simple yet effective strategies to identify winning tickets by looking at the loss landscape and latent features in DNNs. We conduct numerical experiments by collaborating the two proposed methods purifying data and existing robust methods for CIFAR-10 and CIFAR-100. The results support that our framework consistently and remarkably improves performance.

Introduction: The Challenge of Label Noise

Deep Neural Networks (DNNs) are notoriously good at memorizing. While this is great for learning complex patterns, it’s a liability when training data contains "label noise"—images tagged with the wrong category (e.g., a dog labeled as a cat). DNNs will eventually memorize these errors, which ruins their ability to generalize to new, clean data.

Existing solutions usually fall into "noise-cleansing," where the model tries to identify and ignore noisy samples. Most current detectors rely on loss values: the idea is that if a model has a high error on a specific image, that image is likely noisy. However, this creates a "chicken and egg" problem: the classifier used to calculate the loss is itself trained on the noisy data, making its error reports biased and unreliable, especially as noise levels increase.

The Winning Ticket Hypothesis

The authors observe that a dataset containing only clean instances from an original noisy dataset leads to better optima than the original dataset, even with fewer data points. This leads to the winning ticket hypothesis: regardless of the robust training method used, the best performance is achieved when the model is trained on the "winning ticket"—the subset of truly clean samples. To identify these winning tickets, the researchers developed FINE (Filtering Noisy Instances via their Eigenvectors).

FINE: Filtering via Eigenvectors

Instead of looking at how "wrong" a prediction is (loss), FINE looks at where the data sits in the network's internal representation (the penultimate layer). This derivative-free approach doesn't require calculating gradients or relying on the final linear classifier, making it more robust to the "corruption" of the model during training.

The proposed framework: (a) shows the general noise-cleansing pipeline, while (b) details the FINE strategy using eigen decomposition and Gaussian Mixture Models.

The core algorithm works as follows:

  1. Feature Extraction: For each class, the model collects the latent representation vectors of all images assigned to that class.
  2. Gram Matrix & Eigen Decomposition: It constructs a "gram matrix" of these features and performs eigen decomposition to find the principal components (eigenvectors).
  3. Alignment Scoring: The "FINE score" is calculated based on how well an individual image's representation aligns with the first eigenvector (the principal component) of that class.
  4. Clustering: Clean data tends to align strongly with this principal component, while noisy data is scattered. The authors fit a Gaussian Mixture Model (GMM) to these alignment scores to automatically draw a boundary between clean and noisy samples.

Experimental Evaluation

The authors evaluated FINE across several standard and real-world benchmarks, including CIFAR-10/100 with synthetic noise and Clothing1M, a massive real-world dataset containing 1 million images with naturally occurring label noise.

Key Results: * Superior Accuracy: On CIFAR-10 with 80% symmetric noise, FINE achieved 74.2% accuracy compared to Co-teaching’s 66.3%. * Real-World Performance: On the Clothing1M dataset, the FINE-enhanced "F-DivideMix" reached 74.37% accuracy, setting a new high bar for the benchmark. * Scalability: Eigen decomposition can be computationally expensive, but the authors found that you can estimate the principal eigenvector using only 1% of the data and still achieve a cosine similarity of 0.99 with the "true" eigenvector.

Conclusion

FINE provides a theoretically grounded and highly effective way to "clean" datasets. By focusing on latent dynamics rather than loss values, it identifies the "winning ticket" of clean data more reliably, significantly boosting the performance of models trained on noisy real-world data. Future work may extend this eigenvector-based approach to anomaly detection and out-of-distribution (OOD) detection.

Summary updated

Abstract

Modern deep neural networks (DNNs) become frail when the datasets contain noisy (incorrect) class labels. Many robust techniques have emerged via loss adjustment, robust loss function, and clean sample selection to mitigate this issue using the whole dataset. Here, we empirically observe that the dataset which contains only clean instances in original noisy datasets leads to better optima than the original dataset even with fewer data. Based on these results, we state the winning ticket hypothesis: regardless of robust methods, any DNNs reach the best performance when trained on the dataset possessing only clean samples from the original (winning ticket). We propose two simple yet effective strategies to identify winning tickets by looking at the loss landscape and latent features in DNNs. We conduct numerical experiments by collaborating the two proposed methods purifying data and existing robust methods for CIFAR-10 and CIFAR-100. The results support that our framework consistently and remarkably improves performance.
BibTeX
@misc{kim2021winning,
  author = {Taehyeon Kim and Jongwoo Ko and Jinhwan Choi and Sangwook Cho and Se-Young Yun},
  title = {{Winning Ticket in Noisy Image Classification.}},
  year = {2021},
  eprint = {2102.11628},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.11628},
}
Figure