Handling noisy annotations in deep supervised learning

Ichraq Lemghari, Sylvie Le Hégarat‐Mascle, Emanuel Aldea, Jennifer Vandoni2023 · DOI

Abstract: Non-destructive testing (NDT) is employed by companies to assess the features of a material, in order to identify some variations or anomalies in its properties without causing any damage to the original object. In this context of industrial visual inspection, the help of new technologies and especially deep supervised learning is nowadays required to reach a very high level of performance. Data labelling, that is essential to reach such performance, may be fastidious and tricky, and only experts can provide the labelling of the material possible defects. Considering classification problems, this paper addresses the issue of handling noisy labels in datasets. We will first present the existing works related to the problem, our general idea of how to handle it, then we will present our proposed method in detail along with the obtained results that reach more than 0.96 and 0.88 of accuracy for noisified MNIST and CIFAR-10 respectively with a 40% noise ratio. Finally, we present some potential perspectives for future works.

Introduction

Deep learning models typically rely on massive, accurately labeled datasets to achieve high performance. However, in specialized fields like industrial Non-Destructive Testing (NDT), obtaining perfect labels is often unfeasible. Labeling is expensive, time-consuming, and even experts can disagree or make mistakes due to the inherent ambiguity of material defects. Standard neural networks are notorious for "memorizing" these errors, eventually fitting the noise rather than the underlying patterns. This research presents a robust two-step framework for training deep learning models on datasets plagued by noisy or incorrect labels, leveraging Evidential Convolutional Neural Networks (ECNNs) to make "set-valued" decisions.

Proposed Methodology

The authors propose a two-stage strategy that moves away from traditional "hard" classification—where a model must pick exactly one class—toward a more flexible "imprecise" approach based on belief function theory.

  1. Set-Valued Classification: The standard Softmax layer is replaced with an evidential layer and a utility layer. This setup allows the model to output a set of classes when it is uncertain, following the philosophy that it is "better to be imprecise than wrong."
  2. Noise Identification: By analyzing the "mass" assigned to different class sets, the model distinguishes between "clean" samples (assigned to a single class) and "noisy" samples (assigned to a set of classes).
  3. Relabeling and Fine-tuning: Once noisy samples are identified, they are assigned "soft labels" (probability distributions over the predicted set). The model is then fine-tuned using this corrected dataset.
The proposed two-step workflow: from noisy input to set-valued identification and final soft-label relabeling.

Experimental Setup

The method was evaluated using two classic computer vision benchmarks, artificially modified to simulate realistic label corruption:

  • MNIST: Handwritten digits, where noise was introduced by swapping labels between visually similar classes like '1' and '7'.
  • CIFAR-10: Natural images, with noise introduced between similar categories like 'Cat' and 'Dog'.

Noise ratios ranging from 10% to 40% were tested, specifically targeting samples that were already difficult for a baseline model to predict accurately.

Key Insights: The Power of Imprecision

A core finding of this work is that forcing a model to choose a single class for a noisy sample introduces significant bias. By allowing the model to express uncertainty through set-valued outputs, it avoids learning incorrect associations during the initial training phase.

Visualization of decision regions based on belief masses, allowing for 'set-valued' classification in uncertain regions.

Furthermore, the authors found that a specific quantity derived from belief masses creates a clear distributional difference between clean and noisy samples. This allows for an empirical method to set thresholds for relabeling. Unlike global statistical correction methods, this approach evaluates each sample individually to decide if it requires correction.

Results

The proposed method consistently outperformed both a standard CNN and the well-known T-forward correction method across all noise levels. The improvement is particularly robust as the noise ratio increases.

Dataset Noise Ratio Standard CNN Accuracy Proposed Method Accuracy
MNIST 40% 92.32% 96.44%
CIFAR-10 40% 87.01% 88.52%

Conclusion and Future Work

While the current framework demonstrates significant gains, several avenues for future research remain. The authors aim to refine belief modeling to handle "intermediate" sets more effectively (e.g., subsets of 3 classes in a 10-class problem). Additionally, they plan to integrate the utility layer directly into the training loss function to create a more streamlined, end-to-end process. Despite the added mathematical complexity of evidential layers, the approach offers a powerful solution for real-world applications where data quality is imperfect.

Summary updated

Abstract

Non-destructive testing (NDT) is employed by companies to assess the features of a material, in order to identify some variations or anomalies in its properties without causing any damage to the original object. In this context of industrial visual inspection, the help of new technologies and especially deep supervised learning is nowadays required to reach a very high level of performance. Data labelling, that is essential to reach such performance, may be fastidious and tricky, and only experts can provide the labelling of the material possible defects. Considering classification problems, this paper addresses the issue of handling noisy labels in datasets. We will first present the existing works related to the problem, our general idea of how to handle it, then we will present our proposed method in detail along with the obtained results that reach more than 0.96 and 0.88 of accuracy for noisified MNIST and CIFAR-10 respectively with a 40% noise ratio. Finally, we present some potential perspectives for future works.
BibTeX
@misc{lemghari2023handling,
  author = {Ichraq Lemghari and Sylvie Le Hégarat‐Mascle and Emanuel Aldea and Jennifer Vandoni},
  title = {{Handling noisy annotations in deep supervised learning}},
  year = {2023},
  doi = {10.1117/12.2692547},
}
Figure