Emphasis Regularisation by Gradient Rescaling for Training Deep Neural Networks with Noisy Labels

Xinshao Wang, Yang Hua, Elyor Kodirov, Neil Martin Robertson

TL;DR

This paper introduces Derivative Manipulation (DM), a novel method for training deep neural networks on datasets with noisy labels and imbalances. DM directly modifies gradients to assign weights to examples, offering a more flexible and unified approach to example weighting and loss function design, leading to improved robustness and performance across various tasks.

What Problem Does This Paper Address?

Training deep neural networks on real-world datasets often involves dealing with two major challenges: noisy labels (incorrectly labeled data points) and imbalanced datasets (where some classes have significantly fewer examples than others). These issues can severely degrade a model's performance and generalization ability. Existing approaches typically tackle these problems by either designing specialized loss functions or by explicitly weighting individual training examples. However, these methods can be complex, and their effectiveness is often limited. The authors highlight a fundamental connection: loss functions inherently perform example weighting through their gradients. They propose DM to directly manipulate these gradients, offering a more unified and flexible framework that bypasses the need for differentiable loss functions and allows for more direct control over example emphasis.

Methods

The core of DM is the manipulation of the derivative magnitude of the loss function with respect to the model's logits. Instead of deriving example weights from a specific loss function, DM directly defines an "emphasis density function" (EDF) based on the derivative's magnitude. This EDF dictates how much each example should contribute to the model's update.

Key aspects of the method include:

  • Derivative Manipulation (DM): The central idea is to directly modify the gradient signal for each example. This is achieved by scaling the derivative of a base loss (like Cross-Entropy) by a custom weighting function.
  • Emphasis Density Function (EDF): This function, derived from the manipulated gradient magnitude, defines the weighting scheme. It's analogous to a probability density function (PDF) and can be designed to emphasize certain types of examples (e.g., easier, harder, or a balance).
  • Emphasis Mode and Variance: The EDF is characterized by its "emphasis mode" (where the weight is highest) and "emphasis variance" (how spread out the weights are). These parameters allow for fine-grained control over which examples are prioritized.
  • Flexibility: DM is designed to be generic, allowing for the derivation of diverse weighting schemes by choosing different EDFs. The paper explores EDFs inspired by common probability distributions like the Normal, Exponential, and Beta distributions.

Datasets

The paper evaluates DM on several standard datasets:

  • CIFAR-10 and CIFAR-100: Used for image classification tasks, with both clean and synthetically generated noisy labels (symmetric and asymmetric).
  • Clothing 1M: A large-scale dataset with real-world noisy labels for image classification.
  • MARS: A dataset for video person re-identification, used to evaluate DM's performance on imbalanced and abnormal examples.
  • IMDB: A movie review dataset used for sentiment analysis, evaluated under label noise and sample imbalance.
  • Vehicles-10: A custom small-scale dataset derived from CIFAR-100 for fine-grained visual categorization, used to test DM's performance on smaller datasets.

Key Insights and Ideas

  • Loss Functions as Implicit Weighting Schemes: The paper convincingly demonstrates that standard loss functions implicitly assign weights to examples through their gradients. DM makes this explicit and controllable.
  • Direct Gradient Manipulation is More Flexible: By directly manipulating gradients, DM bypasses the constraints of differentiable loss functions, allowing for more complex and effective weighting schemes that might be difficult or impossible to achieve with traditional loss design.
  • Emphasis Mode and Variance Control Robustness: The ability to tune the "emphasis mode" (focusing on easier vs. harder examples) and "emphasis variance" is crucial for adapting to different levels of label noise. Higher noise rates benefit from emphasizing easier examples.
  • DM Unifies Loss and Weighting: DM offers a single framework that can replace both custom loss functions and separate example weighting mechanisms, simplifying the training pipeline.
  • Robustness to Noise: DM consistently outperforms standard methods and even specialized noisy label learning techniques across various datasets and noise levels.

Key Results

  • Improved Accuracy on Noisy Datasets: DM achieved state-of-the-art or competitive results on CIFAR-10/100, Clothing 1M, and IMDB datasets under various label noise conditions. For instance, on CIFAR-100 with 40% symmetric noise, DM achieved up to 67.4% accuracy, outperforming CCE (60.4%) and other methods.
  • Effective on Imbalanced Data: DM showed strong performance on the MARS video retrieval dataset, achieving the best mAP score.
  • Robustness to Different Architectures and Optimizers: The paper demonstrates DM's effectiveness with various network architectures (ResNet, GoogLeNet) and optimizers (SGD, Adam).
  • Label Correction Feasibility: The study showed that retraining with DM on relabeled data (using model predictions) significantly improved accuracy, demonstrating DM's utility in label correction. For example, on CIFAR-10 with 40% noise, label correction with DM yielded an improvement of +2%.

Limitations and Future Work

  • Hyperparameter Tuning: While DM offers flexibility, finding the optimal emphasis mode ($\lambda$) and variance ($\beta$) can still require careful tuning, especially for new datasets. The paper suggests that higher noise rates generally benefit from larger $\lambda$ values.
  • Adaptive Emphasis: The current approach often relies on pre-defined or searched values for emphasis parameters. The authors suggest future work on adaptively setting these parameters based on dataset characteristics and training progress.
  • Derivative Direction: The current DM focuses on manipulating derivative magnitude. The paper acknowledges that manipulating derivative direction could be another avenue for future research, especially for losses with different derivative characteristics.
  • Computational Cost: While DM is presented as a unified approach, the explicit calculation and manipulation of gradients might introduce some computational overhead, though it's generally presented as efficient.

Summary updated

Abstract

Real-world large-scale datasets usually contain noisy labels and are imbalanced. Therefore, we propose derivative manipulation (DM), a novel and general example weighting approach for training robust deep models under these adverse conditions. DM has two main merits. First, loss function and example weighting are common techniques in the literature. DM reveals their connection (a loss function does example weighting) and is a replacement of both. Second, despite that a loss defines an example weighting scheme by its derivative, in the loss design, we need to consider whether it is differentiable. Instead, DM is more flexible by directly modifying the derivative so that a loss can be a non-elementary format too. Technically, DM defines an emphasis density function by a derivative magnitude function. DM is generic in that diverse weighting schemes can be derived. Extensive experiments on both vision and language tasks prove DM's effectiveness.
BibTeX
@misc{wang2019emphasis,
  author = {Xinshao Wang and Yang Hua and Elyor Kodirov and Neil Martin Robertson},
  title = {{Emphasis Regularisation by Gradient Rescaling for Training Deep Neural Networks with Noisy Labels}},
  year = {2019},
  eprint = {1905.11233},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.11233},
}
Figure