TL;DR
This paper presents a comprehensive scoping review of deep learning techniques for handling noisy labels in medical prediction tasks. It highlights that while the medical community is adopting advanced methods, the practical adoption of label noise management is still limited, urging researchers to consider noisy labels as a standard element in medical research.
What Problem Does This Paper Address?
Deep learning (DL) models have shown remarkable success in various medical domains, often achieving expert-level performance. However, the performance of these DL models is highly dependent on the quality of the training data. In medical research, data often suffers from "noisy labels," which can arise from various sources such as inter-expert variability, the inherent complexity of diagnoses, or automated label generation processes. Despite the significant impact of noisy labels on model performance, this issue has been largely overlooked in medical machine learning studies. This paper aims to bridge this gap by providing a comprehensive overview of label noise management in deep learning for medical prediction problems, covering its sources, impact, detection, handling, and evaluation.
Methods
This study employs a scoping review methodology, adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The researchers searched four major databases (PubMed, IEEE Xplore, Google Scholar, and Semantic Scholar) using specific keywords related to noisy labels and medical applications. The review focused on English-language articles published between 2016 and 2023, including both conference papers and journal articles, with an additional consideration for relevant preprint articles. The selection process involved reviewing articles for the source of label noise, problem space, medical problem, label noise detection and handling methods, the need for clean datasets, backbone models, evaluation methods, and publication year. Review articles, non-medical articles, image segmentation problems, and data/feature denoising problems were excluded.
Datasets
The paper does not introduce new datasets. Instead, it reviews existing research that utilizes various medical datasets. Table 1 provides a detailed summary of the included studies, listing the specific datasets used in each. Examples of modalities include MRI, X-ray, CT, OCT, WSI (Whole Slide Images), and fundus images. Diseases covered range from brain diseases and skin cancer to COVID-19 and various other conditions. The paper notes that Chest X-ray and skin cancer datasets are among the most extensively explored due to data availability.
Key Insights and Ideas
- Pervasive Issue, Limited Adoption: Despite the inherent ambiguity in medical diagnoses and the reliance on automated or multi-expert labeling, leading to prevalent noisy labels, the adoption of explicit label noise management techniques in mainstream medical research remains limited.
- Medical DL is "Up-to-Date": The medical community is generally keeping pace with the broader deep learning community in terms of adopting techniques for handling noisy labels, with most reviewed methods being evaluated on medical data.
- Problem-Centric Approach is Key: A comprehensive approach to noisy label management should encompass not only detection and handling but also a thorough understanding of the problem space, including the sources and impact of noise.
- Iterative Refinement is Beneficial: Applying noise-handling techniques iteratively often leads to improved results, with the noise level tending to decrease with each iteration.
- Prioritize Refinement over Filtering: Due to the inherent uncertainty in noise detection, reweighting or label refinement methods are generally preferred over simply filtering out potentially noisy labels.
- Instance-Based Noise Simulation is Ideal but Rare: While instance-based noise injection is the most compelling simulation for label noise, it is rarely applied in practice.
Key Results
The review identified 60 papers that met the inclusion criteria, published between 2016 and 2023. These papers investigated various aspects of label noise in medical research, including its sources, impact, detection, handling techniques, and evaluation. The study categorizes label noise detection methods into model-based detection (e.g., high loss function, high model uncertainty, clustering-based) and external information-based methods (e.g., machine-extracted labels, pseudo-labels, expert agreement). For handling label noise, the paper categorizes techniques into label refinement and filtering, loss and architecture change, loss and model robustness improvement, and training paradigm change. The impact of label noise is shown to be substantial, with studies demonstrating that even a small percentage of injected noise can significantly increase generalization error and affect prediction accuracy.
Limitations and Future Work
The paper acknowledges that while many studies have attempted to address label noise, their suggested approaches are often difficult to implement and may lack broad applicability. Several directions for future research are proposed:
- Distinguishing Noise from Hard Examples: Further research is needed to differentiate between truly noisy instances and genuinely hard-to-learn examples, which can be addressed by considering instances with high loss or model uncertainty.
- Cross-Disciplinary Investigations: Learning from noisy labels is closely linked to other domains like semi-supervised learning, active learning, and regularization. Cross-disciplinary research could offer new insights.
- Backbone Model Sensitivity: More studies are needed to compare different backbone models' sensitivity and robustness to label noise to guide architecture selection.
- Method Comparison and Combination: There is a lack of systematic reviews or benchmark studies that compare the performance of different methods and explore effective ways to combine them.
- Thorough Discussion of Detection Methods: Label noise detection methods have not been thoroughly discussed in existing literature, and further research in this area could aid in handling noise and assessing evaluation data quality.
Summary updated