Improving generalization of deep fault detection models in the presence of mislabeled data

Katharina Rombach, Gabriel Michau, Olga Fink

Abstract

Mislabeled samples are ubiquitous in real-world datasets as rule-based or expert labeling is usually based on incorrect assumptions or subject to biased opinions. Neural networks can "memorize" these mislabeled samples and, as a result, exhibit poor generalization. This poses a critical issue in fault detection applications, where not only the training but also the validation datasets are prone to contain mislabeled samples. In this work, we propose a novel two-step framework for robust training with label noise. In the first step, we identify outliers (including the mislabeled samples) based on the update in the hypothesis space. In the second step, we propose different approaches to modifying the training data based on the identified outliers and a data augmentation technique. Contrary to previous approaches, we aim at finding a robust solution that is suitable for real-world applications, such as fault detection, where no clean, "noise-free" validation dataset is available. Under an approximate assumption about the upper limit of the label noise, we significantly improve the generalization ability of the model trained under massive label noise.

Introduction

In industrial settings, fault detection models often rely on labels generated by "rules of thumb" or subjective expert assessments. These labels are frequently inaccurate, leading to datasets with significant "label noise." Standard deep learning models are prone to memorizing these errors, which degrades their performance on real-world, unseen data. While existing robust training methods often require a small, perfectly clean validation set for hyperparameter tuning, such datasets are rarely available in industrial contexts. This paper presents a framework that achieves robustness even when both training and validation data are noisy.

Methodology

The proposed framework consists of two primary stages designed to prevent the model from overfitting to incorrect labels.

The two-step framework: (1) Outlier Detection via gradient analysis and (2) Data Modification using Adapted Mixup strategies.

Step 1: Outlier Detection (OD)

Rather than relying on final model predictions, the framework analyzes the gradients generated during training. Samples that force the model to make large, inconsistent updates in its internal hypothesis space are flagged as outliers. By using the Interquartile Range (IQR) of mean gradients per layer, the system automatically sets detection thresholds without human intervention.

Step 2: Data Modification (Adapted Mixup)

Once outliers are identified, the training data is modified to mitigate their impact. The authors introduce Adapted Mixup, which applies heavier interpolation to flagged outliers. This "softens" the labels, preventing the network from memorizing them as absolute truths. Strategies include deleting outliers, mixing them with non-outliers, or relabeling them by blending the original noisy label with the model's current prediction.

Experimental Results

The framework was tested on the MNIST dataset (repurposed for binary classification) and a real-world Building Defect Detection (BDD) time-series dataset. Symmetric label noise was injected at levels ranging from 10% to 40%.

Detection of mislabeled samples over training epochs for various noise levels in the MNIST dataset, demonstrating the effectiveness of the gradient-based outlier detection.

As illustrated in the outlier detection analysis, the framework successfully identifies a high proportion of mislabeled samples as training progresses. This early detection is crucial for preventing memorization.

Key performance highlights include:

  • MNIST (40% Noise): The proposed method achieved 93% accuracy, compared to just 62% for standard training (Empirical Risk Minimization).
  • BDD Time-Series (40% Noise): The framework reached 83% accuracy, while standard training performance dropped to 63%.
  • Robustness to Noise Estimation: The model is highly resilient to overestimating the noise level. Assuming a higher noise limit than actually exists still yields strong generalization, whereas underestimation leads to poorer results.

Conclusion

By leveraging gradient updates to identify noisy labels early, this two-step framework provides a robust solution for industrial fault detection. It eliminates the need for clean validation data and significantly improves model generalization under high-noise conditions, making it a viable tool for real-world deployment.

Summary updated

Abstract

Mislabeled samples are ubiquitous in real-world datasets as rule-based or expert labeling is usually based on incorrect assumptions or subject to biased opinions. Neural networks can "memorize" these mislabeled samples and, as a result, exhibit poor generalization. This poses a critical issue in fault detection applications, where not only the training but also the validation datasets are prone to contain mislabeled samples. In this work, we propose a novel two-step framework for robust training with label noise. In the first step, we identify outliers (including the mislabeled samples) based on the update in the hypothesis space. In the second step, we propose different approaches to modifying the training data based on the identified outliers and a data augmentation technique. Contrary to previous approaches, we aim at finding a robust solution that is suitable for real-world applications, such as fault detection, where no clean, "noise-free" validation dataset is available. Under an approximate assumption about the upper limit of the label noise, we significantly improve the generalization ability of the model trained under massive label noise.
BibTeX
@misc{rombach2020improving,
  author = {Katharina Rombach and Gabriel Michau and Olga Fink},
  title = {{Improving generalization of deep fault detection models in the presence of mislabeled data}},
  year = {2020},
  eprint = {2009.14606},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.14606},
}
Figure