Robust Learning from Subjective and Noisy Clinical Labels

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2262 papers283 conceptsfrozen 2026-10-04

In most machine learning benchmarks, "ground truth" is treated as an objective reality. A pixel in ImageNet either belongs to a cat or it doesn't. However, in clinical medicine, the ground truth is often a moving target. Whether a mammogram shows a "malignant" lesion or a "suspicious" one depends heavily on the individual radiologist’s experience, the specific clinical guidelines used, and even the quality of the display hardware.

This inherent inter-observer variability creates a fundamental challenge: when experts disagree, the labels used to train AI models become "noisy." If a model is forced to treat these inconsistent labels as absolute truth, it risks overfitting to human error rather than learning the underlying pathology. Researchers are now developing specialized architectures and training strategies to acknowledge, filter, and even correct this clinical noise.

The Myth of the Gold Standard

The core problem in medical AI is that human-derived labels are frequently inconsistent. As Charles Lu and colleagues note in An Overview and Case Study of the Clinical AI Model Development Life Cycle for Healthcare Systems, generating medical ground truth is resource-intensive and plagued by high inter-annotator variability.

This variability isn't just random error; it is often systemic. In Divergent Domains, Convergent Grading: Enhancing Generalization in Diabetic Retinopathy Grading, Sharon Chokuwa highlights that experts often provide conflicting grades for the same image. When models are trained on these "leaky" reference standards, they can suffer from what Visar Berisha calls "systemic overoptimism"—performing well on paper by mimicking noisy labels but failing to provide reliable diagnostic value in a real clinic.

Filtering Noise via Model Dynamics

One intuitive way to handle noisy labels is to exploit the way deep neural networks learn. Models typically learn simple, consistent patterns (the "signal") early in training, while they only begin to "memorize" complex, inconsistent outliers (the "noise") much later.

Konstantinos Nikolaidis and others leveraged this in Using Under-trained Deep Ensembles to Learn Under Extreme Label Noise. Their framework, Evaluation Probabilities-based Distillation (EPD), uses an ensemble of "under-trained" models to identify which labels are likely incorrect. By stopping training before the models can memorize the noise, the ensemble generates "soft labels" that are cleaner and more representative of the true biological state than the original human annotations.

Similarly, Sebastian Gündel and Florin C. Ghesu demonstrated in Multi-task Learning for Chest X-ray Abnormality Classification on Noisy Labels that filtering out low-confidence predictions—which correlate strongly with radiologist disagreement—can significantly boost model performance, in their case raising AUC from 0.883 to 0.945.

Quantifying Ambiguity

Instead of just filtering noise, some researchers argue that models should explicitly represent the uncertainty inherent in clinical data. In Quantifying and leveraging predictive uncertainty for medical image assessment, Florin C. Ghesu and colleagues used Dempster-Shafer theory to allow models to "self-clean." By predicting an explicit uncertainty score, the model can ignore highly ambiguous or noisy samples during training, ensuring that the final decision boundaries are built on the most reliable data points.

This shift from "hard" labels to "probabilistic" or "ordinal" labels is crucial. Gregor N. Ramien and Simon Köhl addressed this in Reg R-CNN: Lesion Detection and Grading Under Noisy Labels. They argued that treating medical grades (like tumor stages) as discrete classes ignores the fact that a "Grade 2" is closer to a "Grade 3" than a "Grade 5." By using a regression head instead of a classification head, their model became more robust to the "jitter" of annotator disagreement.

Iterative Self-Correction

Perhaps the most ambitious approach is to allow the model to correct its own training data. Jiacheng Li and Ruirui Li proposed a self-relabeling framework in Self-relabeling for noise-tolerant retina vessel segmentation through label reliability estimation.

The intuition is simple: as a model becomes more expert during training, its own predictions may eventually become more reliable than the noisy, manual labels it started with. By estimating the "reliability" of each label based on historical model predictions, the system can iteratively update the training set, effectively "cleaning" the ground truth as it learns. This approach is particularly effective for complex structures like retinal vessels, where human annotators frequently miss thin branches or introduce inconsistencies.

Conclusion

The research direction represented by these papers marks a transition from "naive" supervised learning to "noise-aware" clinical AI. By acknowledging that human experts are not infallible, researchers like Yuan Ma (with LNMBench) and Kumar Abhishek (with DermaMNIST-C) are building a foundation for AI that is robust enough to handle the messy, subjective reality of the hospital floor. The goal is no longer just to match the human label, but to find the biological truth hidden within the noise.

Papers in this version

Showing 60 of 262 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 347 papers
  2. Version 1 · 2026-10-04 · 262 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_inter_observer_variability_label_noise_d5948722,
  title = {inter-observer variability (label noise)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inter-observer-variability-label-noise/rdr_d594872204e348f28eb27bf6824c5d7a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_robust_learning_from_subjective_and_nois_d5948722_v1,
  title = {Robust Learning from Subjective and Noisy Clinical Labels},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/robust-learning-from-subjective-and-noisy-clinical-labels/rdr_d594872204e348f28eb27bf6824c5d7a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}