Feature Noise Boosts DNN Generalization under Label Noise

Lu Zeng, Xuan Chen, Xiaoshuang Shi, Heng Tao ShenIEEE TNNLS 2023 · OpenReview

TL;DR

This paper introduces a simple yet effective "feature noise" (FN) method that adds noise to input features during training to significantly improve how well Deep Neural Networks (DNNs) generalize when their training data contains noisy labels. Theoretically and empirically, they show that this noise injection helps constrain the model's generalization bound, leading to better performance, especially under high label noise conditions.

What Problem Does This Paper Address?

Training deep neural networks (DNNs) is often hampered by "label noise," where some training data samples are incorrectly labeled. This is a significant problem because over-parameterized DNNs can easily memorize these noisy labels, leading to poor performance on unseen data (poor generalization). Existing methods to combat label noise fall into two main categories: denoising-based methods (which try to identify and correct mislabeled samples) and robustness-based methods (which aim to make the model inherently more resistant to noise). While these methods have shown promise, denoising approaches can be complex and require clean samples, and robustness methods often lack a clear theoretical understanding of why they work. This paper tackles the challenge of improving DNN generalization under label noise by proposing a novel, theoretically grounded, and simple robustness-based approach.

Methods

The core contribution is the Feature Noise (FN) method. This technique involves adding random noise (specifically, Gaussian noise in their experiments) directly to the input features of the training data. The authors propose that this simple augmentation helps DNNs generalize better under label noise.

Their theoretical framework is built around PAC-Bayes generalization bounds. They argue that label noise loosens these bounds, making DNNs more prone to overfitting. Feature noise, on the other hand, is shown to constrain these bounds by imposing an upper limit on the mutual information between the model's weights and the input features. This constraint effectively prevents the model from memorizing noisy labels as easily.

They also conduct application analyses to determine the optimal types and levels of feature noise for effective generalization.

Datasets

The authors evaluated their method on a variety of datasets:

  • MNIST: A standard dataset of handwritten digits.
  • FashionMNIST: A dataset of clothing images.
  • miniImageNet: A subset of the ImageNet dataset, used for image classification.
  • ADNI: A dataset of brain scan images, used for graph-based classification.

Key Insights and Ideas

  • Counterintuitive Benefit: Adding noise to input features, which might seem detrimental, actually boosts generalization in the presence of label noise. This is counterintuitive because supervised learning typically aims to learn from clean, accurate signals.
  • Theoretical Foundation: The paper provides a strong theoretical grounding for the effectiveness of feature noise by linking it to the PAC-Bayes generalization bound. This moves beyond heuristic explanations for why robustness techniques work.
  • Constraining Information Flow: Feature noise acts as a bottleneck, limiting the amount of information about the training data (especially noisy labels) that can be encoded into the model's weights. This is key to preventing memorization.
  • Simplicity and Flexibility: The FN method is remarkably simple to implement and can be applied to various models and datasets without significant computational overhead.
  • Randomness Matters: The effectiveness of feature noise is related to its "randomness." More random noise (like Gaussian noise) tends to yield better results than less random noise (like Gaussian blur) as the noise severity increases.

Key Results

  • Significant Accuracy Gains: The FN method consistently improved accuracy across various datasets (MNIST, FashionMNIST, miniImageNet, ADNI) compared to baseline methods and other state-of-the-art label noise techniques, especially under high label noise rates (e.g., ∆ = 0.8). For instance, on MNIST, they reported an average improvement of 2.73% over the best comparison methods.
  • Tighter Generalization Bounds: Theoretical analysis and experimental visualization (Figure 3) showed that training with feature noise resulted in tighter PAC-Bayes generalization bounds compared to training with label noise alone.
  • Improved Interpretability: Saliency maps (Figure 6) revealed that models trained with feature noise generated clearer and more interpretable patterns, focusing on relevant foreground features rather than background noise.
  • Optimal Noise Level: The study found that there's an optimal level of feature noise; too little or too much can be detrimental, and this optimal level is influenced by the label noise rate.

Limitations and Future Work

  • Theoretical Extension: The authors suggest extending their theoretical framework to encompass other types of label noise, such as pair noise and instance-dependent noise.
  • Relationship to Other Methods: Further theoretical work could elucidate the precise relationship between feature noise and other related methods like dropout and data augmentation.
  • Determining Optimal Noise Threshold: While they provide insights into the optimal randomness and level of feature noise, determining a precise threshold that guarantees favorable generalization remains an area for future exploration.
  • Generalization to Other Noise Types: While the paper focuses on symmetric label noise, exploring the effectiveness of FN on other noise distributions would be valuable.

Summary updated

Abstract

The presence of label noise in the training data has a profound impact on the generalization of deep neural networks (DNNs). In this study, we introduce and theoretically demonstrate a simple feature noise method, which directly adds noise to the features of training data, can enhance the generalization of DNNs under label noise. Specifically, we conduct theoretical analyses to reveal that label noise leads to weakened DNN generalization by loosening the PAC-Bayes generalization bound, and feature noise results in better DNN generalization by imposing an upper bound on the mutual information between the model weights and the features, which constrains the PAC-Bayes generalization bound. Furthermore, to ensure effective generalization of DNNs in the presence of label noise, we conduct application analyses to identify the optimal types and levels of feature noise to add for obtaining desirable label noise generalization. Finally, extensive experimental results on several popular datasets demonstrate the feature noise method can significantly enhance the label noise generalization of the state-of-the-art label noise method.
BibTeX
@article{zeng2023feature,
  author = {Lu Zeng and Xuan Chen and Xiaoshuang Shi and Heng Tao Shen},
  title = {{Feature Noise Boosts DNN Generalization under Label Noise}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2023},
  url = {https://openreview.net/forum?id=2PGCqE4p2N},
}
Figure