L_DMI: A Novel Information-theoretic Loss Function for Training Deep Nets Robust to Label Noise

Yilun Xu, Peng Cao, Yuqing Kong, Yizhou WangNeurIPS 2019 · OpenReview

Introduction

Deep learning thrives on massive, accurately labeled datasets. However, high-quality human annotation is expensive and slow. To scale up, researchers often turn to cheaper alternatives like crowdsourcing or web-scraping, which inevitably introduce "label noise"—instances where, for example, an image of a cat is incorrectly labeled as a dog.

Standard loss functions, like Cross Entropy (CE), are notoriously sensitive to these errors. If a dataset is heavily biased (e.g., 90% of images are "benign"), a distance-based loss might encourage a "meaningless classifier" that simply predicts "benign" for everything to minimize error. While previous research has attempted to fix this, most methods require "auxiliary information," such as a small set of perfectly clean data or an estimate of the "noise transition matrix" (the probability of one class being mistaken for another).

To address this gap, researchers have developed $\mathcal{L}_{\text{DMI}}$, a novel loss function that allows deep neural networks to learn effectively even when their training data is riddled with incorrect labels. By swapping traditional distance-based metrics for a generalized version of mutual information, this approach is provably robust against various noise patterns without requiring any prior knowledge of how the labels were corrupted.

Methods: The Determinant-Based Mutual Information (DMI)

The authors introduce $\mathcal{L}_{\text{DMI}}$, based on a concept called Determinant-based Mutual Information (DMI). Unlike standard Shannon Mutual Information, which measures the reduction in uncertainty, DMI measures the correlation between two random variables (the classifier's predictions and the noisy labels) by looking at the determinant of their joint distribution matrix.

Pipeline for calculating the L_DMI loss function

The core logic of this approach relies on three main pillars:

  1. Information-Theoretic vs. Distance-Based: Instead of measuring how "far" a prediction is from a label, $\mathcal{L}_{\text{DMI}}$ maximizes the information shared between them.
  2. The Determinant Trick: If a classifier is "meaningless" (guessing randomly or always picking one class), its joint distribution matrix with the labels will have a low rank, resulting in a determinant of zero. This naturally penalizes lazy learning strategies.
  3. Relative Invariance: The authors prove that DMI possesses a property called "relative invariance." This means that the ranking of classifiers based on noisy labels is mathematically identical to the ranking based on clean labels. Essentially, the "best" model under noisy conditions is guaranteed to be the "best" model under clean conditions.

Crucially, $\mathcal{L}_{\text{DMI}}$ is a "plug-and-play" solution. It can be applied to any existing classification architecture (ResNet, VGG, etc.) straightforwardly without requiring changes to the model's internal structure or the estimation of a noise transition matrix.

Datasets and Experiments

The researchers evaluated $\mathcal{L}_{\text{DMI}}$ across a diverse range of benchmarks:

  • Computer Vision: Fashion-MNIST, CIFAR-10, and Dogs vs. Cats.
  • Natural Language Processing: The MR (Movie Review) dataset.
  • Real-World Noise: Clothing1M, a massive dataset of 1 million images with real-world label noise scraped from the web, which serves as a rigorous test for practical application.

For the synthetic datasets, they tested three noise patterns: uniform noise, biased noise (flipping rare classes to common ones), and "risk-avoid" noise (flipping common classes to rare ones).

Key Results

Traditional losses often fail when noise is "diagonally non-dominant"—for instance, when an annotator is so lazy they just pick the most popular class regardless of the image. $\mathcal{L}_{\text{DMI}}$ naturally ignores these strategies because a constant output provides zero mutual information.

Accuracy of DMI vs. Cross Entropy (CE) on Fashion-MNIST across varying noise amounts
  • Superiority in High Noise: On datasets like Fashion-MNIST, CIFAR-10, and Dogs vs. Cats, $\mathcal{L}_{\text{DMI}}$ maintained high accuracy even as the noise amount ($r$) approached 1.0. It significantly outperformed Cross Entropy and other robust baselines like Generalized Cross Entropy (GCE), particularly in diagonally non-dominant noise regimes.
  • Real-World Performance: On the Clothing1M dataset, $\mathcal{L}_{\text{DMI}}$ achieved an accuracy of 72.46%, outperforming the standard Cross Entropy (68.94%) and other specialized noise-robust methods like LCCN (71.63%).
  • Theoretical Equivalence: The authors proved that training with $\mathcal{L}_{\text{DMI}}$ on noisy data is theoretically equivalent to training on clean data, plus a constant "noise amount" shift that doesn't affect the optimization path.

Limitations and Future Work

While highly effective, the method does have some limitations. The mathematical proof for $\mathcal{L}_{\text{DMI}}$ assumes that label noise is instance-independent—meaning it depends only on the class (e.g., "dogs are often confused for cats") rather than the specific image (e.g., "this specific blurry dog looks like a cat"). Real-world noise is often instance-dependent.

Additionally, the authors noted that on perfectly clean data, $\mathcal{L}_{\text{DMI}}$ sometimes performs slightly worse than GCE or CE. They suggest that future work could involve hybridizing DMI with other training optimizations to bridge this gap. Finally, the use of determinants and logs can occasionally lead to computational scaling issues during backpropagation, though the authors suggest the log function helps mitigate this in practice.

Summary updated

Abstract

Accurately annotating large scale dataset is notoriously expensive both in time and in money. Although acquiring low-quality-annotated dataset can be much cheaper, it often badly damages the performance of trained models when using such dataset without particular treatment. Various methods have been proposed for learning with noisy labels. However, most methods only handle limited kinds of noise patterns, require auxiliary information or steps (e.g., knowing or estimating the noise transition matrix), or lack theoretical justification. In this paper, we propose a novel information-theoretic loss function, L<em>DMI, for training deep neural networks robust to label noise. The core of L</em>DMI is a generalized version of mutual information, termed Determinant based Mutual Information (DMI), which is not only information-monotone but also relatively invariant. To the best of our knowledge, L<em>DMI is the first loss function that is provably robust to instance-independent label noise, regardless of noise pattern, and it can be applied to any existing classification neural networks straightforwardly without any auxiliary information. In addition to theoretical justification, we also empirically show that using L</em>DMI outperforms all other counterparts in the classification task on both image dataset and natural language dataset include Fashion-MNIST, CIFAR-10, Dogs vs. Cats, MR with a variety of synthesized noise patterns and noise amounts, as well as a real-world dataset Clothing1M.
BibTeX
@inproceedings{xu2019l,
  author = {Yilun Xu and Peng Cao and Yuqing Kong and Yizhou Wang},
  title = {{L\_DMI: A Novel Information-theoretic Loss Function for Training Deep Nets Robust to Label Noise}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=Uj7SfylduF},
}
Figure