Using Under-trained Deep Ensembles to Learn Under Extreme Label Noise

Konstantinos Nikolaidis, Thomas Plagemann, Stein Kristiansen, Vera Goebel, Mohan S. Kankanhalli

Introduction

Training machine learning models on datasets with highly inaccurate labels usually leads to poor generalization, especially in critical fields like healthcare. This paper introduces Evaluation Probabilities-based Distillation (EPD), a technique that uses an ensemble of intentionally under-trained neural networks to filter out label noise and generate clean "soft labels" for training a final, robust model.

The Problem: Label Noise in Critical Domains

Supervised learning typically requires high-quality, accurately labeled data. However, real-world datasets—particularly in healthcare—are often plagued by label noise due to human error, ambiguous data, or incomplete information. Labeling requires expensive expert time, and models trained on corrupted labels can lead to life-threatening misdiagnoses. While existing research often focuses on computer vision, this work addresses the problem in medical time-series sensor data, such as diagnosing sleep apnea from respiratory signals, where noise can reach extreme levels (up to 50%).

Methodology: Evaluation Probabilities-based Distillation (EPD)

EPD exploits a key characteristic of Deep Neural Networks (DNNs): they tend to learn simple, generalized patterns before memorizing complex, noisy labels.

The EPD workflow: (1) Training an ensemble of under-trained models on data subsets, (2) generating out-of-bag Evaluation Probabilities (EP), and (3) distilling these into a final model.

The process consists of three main stages:

  1. Under-Trained Ensemble: A large ensemble of base models (e.g., $K=100$) is initialized. Each model is trained on a random subset of the training data and aggressively under-trained (e.g., stopped after just 4 epochs). This prevents the models from memorizing noisy, incorrect labels while allowing them to capture general decision boundaries.
  2. Out-of-Bag Soft Labels: For each data point, the system averages predictions from the subset of models that did not see that point during training. This "Evaluation Probability" (EP) serves as a denoised soft label.
  3. Knowledge Distillation: The original noisy labels are discarded. A final model is trained using the EPs as ground truth via knowledge distillation.

Experimental Setup and Noise Models

The authors evaluated EPD on medical time-series (Apnea-ECG, MIT-BIH) and standard vision benchmarks (MNIST, SVHN). To simulate realistic and extreme noise, they employed several noise transition matrices to corrupt the training data.

Visualizing label noise patterns: (a) Symmetry noise, (b) Pair noise (confusing adjacent classes), and (c) Alphanumeric noise (simulating human misidentification, e.g., confusing '3' for '8').

Key Results

EPD demonstrated significant robustness compared to baseline models trained directly on noisy data:

  • Extreme Vision Noise: On MNIST with 45% "Pair" noise, baseline accuracy dropped to 56.1%, while EPD maintained 95.0% accuracy.
  • Medical Time-Series: For sleep apnea detection with 45% Pair noise, the baseline model's Kappa score fell to a near-useless 0.103. EPD achieved a Kappa of 0.584.
  • Realistic Human Error: Using alphanumeric misidentification patterns, EPD improved MNIST accuracy from 63.3% to 94.9%.
  • Non-Expert Labeling: EPD proved viable for low-budget scenarios, allowing models trained on non-expert labels to perform nearly as well as those trained on expert data.

Insights and Limitations

A critical finding is that under-training is a feature, not a bug. By stopping early, base models act as natural low-pass filters for noise. The authors also identified that tracking the minimum mean entropy of the EP is a reliable proxy for generalization when a clean validation set is unavailable.

However, EPD is limited by systematic bias. If an annotator consistently mislabels a specific feature space due to a fundamental misunderstanding, the ensemble will learn that incorrect rule. Future work involves combining EPD with unsupervised learning to detect such consistent biases.

Summary updated

Abstract

Improper or erroneous labelling can pose a hindrance to reliable generalization for supervised learning. This can have negative consequences, especially for critical fields such as healthcare. We propose an effective new approach for learning under extreme label noise, based on under-trained deep ensembles. Each ensemble member is trained with a subset of the training data, to acquire a general overview of the decision boundary separation, without focusing on potentially erroneous details. The accumulated knowledge of the ensemble is combined to form new labels, that determine a better class separation than the original labels. A new model is trained with these labels to generalize reliably despite the label noise. We focus on a healthcare setting and extensively evaluate our approach on the task of sleep apnea detection. For comparison with related work, we additionally evaluate on the task of digit recognition. In our experiments, we observed performance improvement in accuracy from 6.7\% up-to 49.3\% for the task of digit classification and in kappa from 0.02 up-to 0.55 for the task of sleep apnea detection.
BibTeX
@article{nikolaidis2021using,
  author = {Konstantinos Nikolaidis and Thomas Plagemann and Stein Kristiansen and Vera Goebel and Mohan S. Kankanhalli},
  title = {{Using Under-trained Deep Ensembles to Learn Under Extreme Label Noise}},
  journal = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2021.3067455},
}
Figure