USDNL: Uncertainty-Based Single Dropout in Noisy Label Learning

Yuanzhuo Xu, Xiaoguang Niu, Jie Yang, Steve Drew, Jiayu Zhou, Ruizhi ChenAAAI 2023 · DOI

Abstract

Deep Neural Networks (DNNs) are susceptible to memorizing noisy labels, which degrades performance on large-scale datasets. Current state-of-the-art methods often employ dual-network architectures to filter noise, doubling computational costs. We present USDNL (Uncertainty-based Single Dropout in Noisy label Learning), a method that leverages epistemic uncertainty for clean sample selection. We theoretically and empirically demonstrate that a single dropout pass is sufficient for effective uncertainty estimation in this context. USDNL achieves state-of-the-art results across various noise types while being nearly twice as fast as dual-network baselines.

Introduction

The prevalence of noisy labels in web-scraped datasets poses a significant challenge for training robust DNNs. While "small-loss selection" heuristics—based on the observation that models learn clean patterns before memorizing noise—are effective, popular implementations like Co-teaching or JoCoR require training two networks simultaneously. This "dual-network" approach is computationally expensive and lacks a rigorous theoretical foundation. USDNL addresses these limitations by using model uncertainty as a more principled selection criterion within a single-network framework.

Methodology

The core of USDNL is the connection between label noise and epistemic uncertainty. Epistemic uncertainty reflects the model's lack of knowledge about specific data points, making it a natural indicator for noisy or out-of-distribution samples.

The Single Dropout Breakthrough

Traditionally, estimating epistemic uncertainty requires multiple Monte Carlo (MC) dropout passes, which is computationally intensive. A key contribution of this work is the theoretical proof that for the purpose of selecting clean samples, a single dropout pass ($N=1$) is mathematically equivalent to multiple passes.

Figure 1: Comparison of different dropout times (1 to 5) on test accuracy and label precision, demonstrating that a single pass is sufficient.

As shown in Figure 1, increasing the number of dropout iterations does not yield significant gains in accuracy or label precision, validating the efficiency of the $N=1$ approach.

Selection and Training

USDNL employs a selection loss ($L_{sl}$) derived from the cross-entropy of a single dropout sub-model. During training, samples with low selection loss are identified as "clean" and used for the primary model update. To prevent the model from eventually memorizing noise, a "warm-up" period is followed by a progressive selection schedule that gradually reduces the ratio of trusted samples.

Experimental Results

The performance of USDNL was evaluated across synthetic noise (CCN), out-of-distribution noise (CIFAR80N-O), instance-dependent noise (IDN), and real-world noise (Clothing1M).

Accuracy and Robustness

USDNL consistently outperforms existing methods across various noise regimes. For instance, on CIFAR-100 with 50% symmetric noise, USDNL achieved 63.42% accuracy, significantly outperforming CDR (47.17%) and JoCoR (43.38%).

Figure 3: Performance comparison across different noise types and levels. USDNL (red) shows superior test accuracy and maintains higher label precision throughout training.

As illustrated in Figure 3, USDNL maintains high label precision throughout the training process, whereas competing methods often suffer from "noise leakage" in later epochs, where noisy samples are incorrectly identified as clean.

Efficiency

By avoiding the dual-network requirement, USDNL is significantly more efficient. Training time benchmarks show it is nearly twice as fast as methods like JoCoR. On a 9-CNN architecture, USDNL required 32.3s per epoch, while JoCoR and CausalNL required 54.7s and 112.4s, respectively.

Conclusion

USDNL provides a theoretically grounded and computationally efficient solution for learning with noisy labels. By demonstrating that a single dropout pass is sufficient for uncertainty-based sample selection, it bridges the gap between high-performance noise filtering and practical training speeds. Future work will explore adaptive dropout rates to further enhance robustness in high-noise, instance-dependent scenarios.

Summary updated

BibTeX
@inproceedings{xu2023usdnl,
  author = {Yuanzhuo Xu and Xiaoguang Niu and Jie Yang and Steve Drew and Jiayu Zhou and Ruizhi Chen},
  title = {{USDNL: Uncertainty-Based Single Dropout in Noisy Label Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i9.26264},
}
Figure