How Does Heterogeneous Label Noise Impact Generalization in Neural Nets?

Bidur Khanal, Christopher Kanan

Abstract

Recent studies in deep learning have extensively explored the impact of label noise on model generalization. However, most research assumes "homogeneous" noise, where errors are uniformly distributed across all classes. In reality, datasets often exhibit "heterogeneous" noise, where some categories are accurately labeled while others are highly corrupted. This paper investigates whether noise in one category "leaks" to degrade performance in clean categories across multi-class, multi-task, and multi-label learning paradigms. Our findings suggest that while neural networks are robust to such noise in standard classification, heterogeneous noise significantly diminishes the performance-boosting benefits of transfer learning in multi-task settings.

Introduction: The Reality of Messy Data

Deep learning models are notoriously data-hungry, often relying on massive datasets labeled via crowdsourcing or automated scraping. These processes inevitably introduce label noise. While homogeneous noise is well-studied, real-world noise is frequently heterogeneous. For instance, a dataset might contain perfectly labeled "airplanes" but unreliable labels for "cats" and "dogs" due to inter-class similarity. A fundamental question arises: does the noise in one category corrupt the model's ability to learn a completely different, clean category?

Conceptual diagram of training with heterogeneous label noise (noisy vs. correct labels) and evaluation on clean data.

Experimental Methodology

The researchers systematically injected noise into popular computer vision benchmarks—MNIST, CIFAR-10, CIFAR-100, and MS-COCO—to observe cross-contamination effects across three paradigms:

  1. Multi-class Classification: Nine out of ten classes were corrupted with varying noise levels while the tenth remained clean. They tested scenarios where noisy samples could or could not be mislabeled as the "clean" class.
  2. Multi-task Learning (MTL): Datasets were split into two tasks. Task A remained clean, while Task B was corrupted to measure the impact on "transfer benefit"—the performance boost Task A typically receives from joint training.
  3. Multi-label Learning: Labels were grouped into corrupted and uncorrupted sets to see if errors in one set (e.g., animal tags) affected the accuracy of others (e.g., vehicle tags) on the same image.
Multi-task learning architecture using CIFAR-100, where the model branches into two 50-class tasks from a shared ResNet-18 backbone.

Key Insights

Compartmentalization of Learning

In standard classification, neural networks demonstrate a surprising ability to "ignore" noise in specific segments of the dataset. The accuracy of clean classes remained stable even when other classes were 80-100% noisy. This suggests that high-level label errors in one category do not necessarily corrupt the shared low-level feature filters used by the entire network.

The Vanishing Transfer Benefit

In Multi-task Learning, a "helper" task usually improves the main task. However, as the helper task becomes noisier, this transfer benefit evaporates. Interestingly, the noise rarely caused "negative interference"; the model essentially "stops listening" to the noisy task and reverts to its single-task performance level rather than performing worse than if trained alone.

Robustness to Overfitting

While models can eventually "memorize" noisy labels if overtrained, this memorization does not immediately destroy the representation of clean classes. The network maintains a degree of separation between the clean and noisy signals.

Key Results

  • Multi-class (CIFAR-10): Accuracy of the uncorrupted class remained flat as noise in other classes scaled from 0% to 100%.
  • Multi-task (CIFAR-100): Task 1's performance jumped from a 75.63% baseline to 79.50% when paired with a clean Task 2. As noise was added to Task 2, Task 1's performance gradually returned to the 75% baseline but did not drop significantly below it.
  • Multi-label (MS-COCO): Mean Average Precision (mAP) for uncorrupted label groups remained consistent regardless of corruption levels in other groups.

Conclusion and Future Directions

This study highlights the inherent robustness of neural networks to heterogeneous label noise in classification tasks. However, the loss of transfer benefits in multi-task settings poses a challenge for leveraging noisy auxiliary data. Future work should explore whether these findings hold for regression tasks and investigate the layer-by-layer impact of noise on feature representations to better understand the mechanisms of this observed robustness.

Summary updated

BibTeX
@inproceedings{khanal2021how,
  author = {Bidur Khanal and Christopher Kanan},
  title = {{How Does Heterogeneous Label Noise Impact Generalization in Neural Nets?}},
  booktitle = {ISVC},
  year = {2021},
  url = {https://openreview.net/forum?id=dDqJhZmGBvL},
}
Figure