Confidence-based Reliable Learning under Dual Noises

Peng Cui, Yang Yue, Zhijie Deng, Jun Zhu

TL;DR

Most research on "noisy data" focuses on either bad labels ($y$-noise) or corrupted images ($x$-noise), but rarely both. This paper introduces a unified framework that uses the predictive confidence of deep ensembles to filter out both types of noise simultaneously. Instead of discarding noisy samples, the authors apply an uncertainty penalty that allows the model to learn from the data without overfitting to its misleading components.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are notoriously "data-hungry" and sensitive. In real-world scenarios, datasets are rarely pristine. You often encounter two types of interference:

  1. Label Noise ($y$-noise): An image of a lion is incorrectly tagged as a "tiger."
  2. Image Noise ($x$-noise): The image itself is blurred, corrupted, or contains a confusing background that makes classification ambiguous.
Examples of label noise and image noise in datasets.

While there are many techniques to handle one or the other, a "naive combination" of these solutions often fails. For instance, image denoising tools struggle with global blur, and label-correction methods can be tricked by corrupted images into making overconfident, incorrect predictions. The authors argue that we need a unified approach that treats $x$-noise and $y$-noise as a joint problem rather than two separate ones.

Methods

The authors propose a workflow based on Deep Ensembles (training multiple models and averaging their outputs). Their method consists of two main stages:

The proposed confidence-based reliable learning framework.

1. The Confidence-based Sample Filter

The framework uses two specific metrics to identify "dirty" data:

  • Label Confidence (L-Con): This is the model's predicted probability for the given label. If the model is very unconfident about the provided label, it’s likely $y$-noise (a mislabeling).
  • Maximum Confidence (M-Con): This is the highest probability the model assigns to any class. If the model isn't confident about any category, the image itself is likely too corrupted or ambiguous to be useful ($x$-noise).

By setting thresholds ($\epsilon_1, \epsilon_2$), the system flags samples as "clean" or "noisy" in parallel.

2. Uncertainty Penalty

Standard training forces a model to minimize "cross-entropy loss," essentially pushing it to be 100% certain about a label. If the label is wrong, the model overfits to that error.

To combat this, the authors introduce a dual-objective loss function:

  • For Clean Data: Use standard cross-entropy to learn the correct features.
  • For Noisy Data: Instead of ignoring these samples, the model is trained to minimize model uncertainty (measured via Mutual Information). This encourages the ensemble members to reach a consensus of "I don't know," preventing the model from learning misleading patterns while still acknowledging the data exists.

Datasets

The researchers evaluated their method on three major benchmarks:

  • CIFAR-100 & TinyImageNet: Used for synthetic noise tests. They injected varying levels of "dual noise" (e.g., 40% wrong labels combined with 30% image corruption).
  • WebVision: A massive real-world dataset containing over 2.4 million images crawled from Flickr and Google. This dataset is "naturally" noisy, making it a gold standard for testing robustness in the wild.
  • ILSVRC12 (ImageNet): Used to verify if models trained on noisy data (WebVision) can generalize to clean, human-annotated validation sets.

Key Insights and Ideas

  • M-Con as a Proxy for Aleatoric Uncertainty: The paper highlights that Maximum Confidence is a strong indicator of "data uncertainty." When an image is of poor quality, the model's output distribution becomes "flat" (low M-Con), which is a signal to the filter that the image itself is the problem.
  • Filtering vs. Restoration: Unlike methods that try to "fix" blurry images (which is computationally expensive and often fails), this approach simply identifies them and changes how the model learns from them.
  • The "Aha" Moment on Uncertainty: Most noise-robust methods try to find a "pseudo-label" for noisy data. This paper suggests that if the data is truly dual-noisy, any pseudo-label is a guess. Penalizing uncertainty is a safer, more reliable way to handle "garbage" data without letting it poison the model's weights.

Key Results

The proposed method, dubbed Proposed-LM, consistently outperformed state-of-the-art baselines:

  • Synthetic Dual Noise: On CIFAR-100 with 40% label noise and 30% image noise, the method achieved 66.61% accuracy, significantly higher than standard Deep Ensembles (54.90%) and other robust methods like DYR (50.54%).
  • Real-World Performance: On the WebVision benchmark, the method reached a 76.68% top-1 accuracy, outperforming the next best baseline (DE-CE) by nearly 3%.
  • Robustness: Ablation studies showed the method isn't overly sensitive to the specific thresholds chosen for the filters, making it practical for real-world deployment where the exact noise ratio is unknown.

Limitations and Future Work

  • Computational Cost: Because the method relies on Deep Ensembles (training 5+ models), it requires more memory and compute power than a single-model approach. The authors suggest exploring "cheaper" ensemble methods like BatchEnsemble in the future.
  • Coarse X-Noise Handling: The current filter treats all image noise the same. The authors note that a more fine-grained approach that distinguishes between different types of image corruption (e.g., blur vs. background clutter) could further improve performance.
  • Hyperparameter Tuning: While robust, the method still introduces two new thresholds ($\epsilon_1, \epsilon_2$) that need to be set, though the paper provides guidance on how to do this effectively.

Summary updated

Abstract

Deep neural networks (DNNs) have achieved remarkable success in a variety of computer vision tasks, where massive labeled images are routinely required for model optimization. Yet, the data collected from the open world are unavoidably polluted by noise, which may significantly undermine the efficacy of the learned models. Various attempts have been made to reliably train DNNs under data noise, but they separately account for either the noise existing in the labels or that existing in the images. A naive combination of the two lines of works would suffer from the limitations in both sides, and miss the opportunities to handle the two kinds of noise in parallel. This works provides a first, unified framework for reliable learning under the joint (image, label)-noise. Technically, we develop a confidence-based sample filter to progressively filter out noisy data without the need of pre-specifying noise ratio. Then, we penalize the model uncertainty of the detected noisy data instead of letting the model continue over-fitting the misleading information in them. Experimental results on various challenging synthetic and real-world noisy datasets verify that the proposed method can outperform competing baselines in the aspect of classification performance.
BibTeX
@inproceedings{cui2022confidence,
  author = {Peng Cui and Yang Yue and Zhijie Deng and Jun Zhu},
  title = {{Confidence-based Reliable Learning under Dual Noises}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=7fGIR2oIHTl},
}
Figure