Estimating Noise Transition Matrix with Label Correlations for Noisy Multi-Label Learning

Shikun Li, Xiaobo Xia, Hansong Zhang, Yibing Zhan, Shiming Ge, Tongliang LiuNeurIPS 2022 · OpenReview

Executive Summary

This paper introduces a novel method for training multi-label classifiers on datasets with "noisy" (incorrect or missing) labels by leveraging the natural correlations between different classes. By moving away from the restrictive need for "anchor points"—perfectly labeled examples that are exceedingly rare in multi-label data—the authors provide a more robust way to estimate how labels get flipped, leading to significantly better classification performance.

The Challenge of Noisy Multi-Label Data

In real-world applications, data is inherently messy. We frequently encounter "multi-label" data (e.g., an image containing a "dog," a "tree," and a "park"), but the labels are often incorrect or missing because they were collected cheaply or automatically. To train a reliable model on this noisy data, researchers often use a noise transition matrix—a mathematical bridge that describes the probability of a "clean" label being flipped into a "noisy" one.

The primary challenge lies in estimating this matrix. Most existing methods for single-label (multi-class) problems rely on "anchor points"—specific data instances that belong to a class with 100% certainty. In multi-label settings, finding these anchor points is incredibly difficult due to the sheer number of label combinations and the severe imbalance between positive and negative examples. This paper tackles the problem of estimating the transition matrix without needing these elusive anchor points.

Proposed Methodology: Exploiting Label Correlations

The authors propose a two-stage estimator that exploits label correlations (the fact that certain labels, like "fish" and "water," often appear together in the real world).

  1. Sample Selection (The Memorization Effect): Deep neural networks tend to "memorize" clean, easy examples before they start fitting noisy ones. The authors use this trait to select a subset of examples that are highly likely to have clean labels for a specific class.
  2. Co-occurrence Estimation: They calculate two key probabilities: the probability of two noisy labels appearing together in the full dataset, and the probability of a noisy label appearing when a "clean" label (from the selected subset) is present.
  3. Bilinear Decomposition: The authors prove mathematically that the transition matrix is "identifiable" (meaning it can be uniquely determined) by looking at the mismatch between these noisy and clean label correlations. They frame the estimation as a simple bilinear decomposition problem, which can be solved efficiently.

Crucially, this method doesn't require the model to perfectly fit the noisy data or find perfect anchor points; it merely needs to observe how labels tend to co-occur.

Core Innovations

  • Correlations as a Signal: The core innovation is realizing that noise disrupts natural label correlations. If "fish" and "sky" suddenly show a high correlation in a noisy dataset, that "mismatch" provides enough statistical information to reverse-engineer the noise process.
  • Moving Beyond Anchor Points: By proving the "identifiability" of the matrix through correlations, the authors show that the strict (and often unrealistic) requirements of previous noise-learning research can be relaxed for multi-label tasks.
  • Robustness to Bias: Even if the "sample selection" stage is slightly biased (e.g., picking only the "easiest" examples of a class), the authors demonstrate mathematically and empirically that this does not lead to large errors in the final transition matrix estimation.

Performance Highlights

The researchers evaluated their method on three widely used multi-label benchmarks (Pascal-VOC 2007, Pascal-VOC 2012, and MS-COCO), injecting various types of synthetic noise such as missing labels, extra labels, and random flips.

Superior Estimation: On the evaluated datasets, the proposed estimator consistently achieved lower error rates in identifying the transition matrix compared to standard baselines like the "T-estimator" and "Dual T-estimator."

Estimation Error Comparison

Improved Accuracy: When the estimated matrix was used to "correct" the loss function during training (a method known as Reweight), the resulting models outperformed 10 other state-of-the-art baselines.

Performance on MS-COCO: On the challenging MS-COCO dataset—which features 80 object classes and complex label combinations—under a severe 60% noise rate, their method achieved a Per-class F1-measure (CF1) of 65.78%. In stark contrast, the standard training approach plummeted to just 7.07%.

Limitations and Future Directions

  • Dependence on Memorization: The method relies heavily on the "memorization effect" of deep learning. It might not be directly applicable to other types of machine learning models (like decision trees) where this effect isn't as well-defined.
  • Frequency Requirements: For the statistical counting to be accurate, each label needs to appear a minimum number of times (the authors suggest at least 50). This could pose a challenge for extremely "long-tailed" datasets where some classes are exceptionally rare.
  • Instance-Independent Assumption: The current model assumes that the probability of a label being flipped depends on the class, but not on the specific features of the image (instance-independent noise). Future work could explore "instance-dependent" noise, which is computationally more complex but even more reflective of real-world annotation errors.

Summary updated

Abstract

In label-noise learning, the noise transition matrix, bridging the class posterior for noisy and clean data, has been widely exploited to learn statistically consistent classifiers. The effectiveness of these algorithms relies heavily on estimating the transition matrix. Recently, the problem of label-noise learning in multi-label classification has received increasing attention, and these consistent algorithms can be applied in multi-label cases. However, the estimation of transition matrices in noisy multi-label learning has not been studied and remains challenging, since most of the existing estimators in noisy multi-class learning depend on the existence of anchor points and the accurate fitting of noisy class posterior. To address this problem, in this paper, we first study the identifiability problem of the class-dependent transition matrix in noisy multi-label learning, and then inspired by the identifiability results, we propose a new estimator by exploiting label correlations without neither anchor points nor accurate fitting of noisy class posterior. Specifically, we estimate the occurrence probability of two noisy labels to get noisy label correlations. Then, we perform sample selection to further extract information that implies clean label correlations, which is used to estimate the occurrence probability of one noisy label when a certain clean label appears. By utilizing the mismatch of label correlations implied in these occurrence probabilities, the transition matrix is identifiable, and can then be acquired by solving a simple bilinear decomposition problem. Empirical results demonstrate the effectiveness of our estimator to estimate the transition matrix with label correlations, leading to better classification performance. Source codes are available at https://github.com/tmllab/Multi-Label-T.
BibTeX
@inproceedings{li2022estimating,
  author = {Shikun Li and Xiaobo Xia and Hansong Zhang and Yibing Zhan and Shiming Ge and Tongliang Liu},
  title = {{Estimating Noise Transition Matrix with Label Correlations for Noisy Multi-Label Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=wNFdSRxPG-},
}
Figure