Addressing label noise for electronic health records: insights from computer vision for tabular data

Jenny Yang, Hagen Triendl, Andrew A. S. Soltan, Mangal Prakash, David A. CliftonBMC Medical Informatics Decis. Mak 2024 · OpenReview

TL;DR

This paper tackles the pervasive problem of noisy labels in Electronic Health Records (EHRs) by adapting computer vision (CV) techniques, traditionally used for image data, to tabular EHR data. The authors demonstrate that these CV-inspired methods significantly improve the robustness and accuracy of machine learning models, especially when dealing with imperfectly labeled healthcare data, offering a promising avenue for more reliable EHR analysis.

What Problem Does This Paper Address?

Electronic Health Records (EHRs) are a treasure trove of patient data, crucial for advancing healthcare through machine learning (ML). However, EHR data is notoriously prone to errors and noise. This noise can stem from various sources: data entry mistakes, inconsistencies, system glitches, and, critically, human errors in labeling during data collection or annotation. This "label noise" is particularly problematic for deep learning (DL) models, which can easily overfit to these incorrect labels, leading to unreliable predictions, biased outcomes, and a general erosion of trust in ML applications within healthcare. While label noise is a known issue, few studies have specifically addressed it within the unique context of EHR data, especially when the data is tabular rather than image-based.

Methods

The core of this research lies in adapting established computer vision (CV) techniques designed to handle noisy labels to the domain of tabular EHR data. The authors explore three specific CV-inspired methods:

  • Label Smoothing: This technique introduces a small amount of uncertainty to the target labels during training. Instead of hard 0s and 1s, it uses values like $1-\epsilon$ for the correct label and $\epsilon/C$ for others (where $C$ is the number of classes). This discourages the model from becoming overly confident in its predictions for any single label.
  • Mix-up: This data augmentation method creates new training examples by taking convex combinations of existing samples and their labels. This encourages smoother decision boundaries and makes the model more robust to noisy labels by preventing it from memorizing specific, potentially incorrect, data points.
  • Neighbor Consistency Regularization (NCR): This method enforces that similar data points in the feature space should have similar predictions, regardless of their potentially noisy labels. It uses cosine similarity to measure feature similarity and a Kullback-Leibler divergence loss to encourage consistent outputs for neighbors.

The study investigates these methods both individually and in combination, applying them to a baseline neural network architecture and an XGBoost model.

Datasets

The primary dataset used for this study is the CURIAL dataset, which comprises anonymized EHR data from four independent UK National Health Service (NHS) Trusts: Oxford University Hospitals NHS Foundation Trust (OUH), University Hospitals Birmingham NHS Trust (UHB), Portsmouth Hospitals University NHS Trust (PUH), and Bedfordshire Hospitals NHS Foundations Trust (BH). This dataset is particularly relevant as it contains data related to COVID-19 diagnosis, a scenario where label noise is plausible due to imperfect test sensitivities and varying testing protocols.

The data was split into training, validation, and test sets. Additionally, external test sets from three independent hospitals were used for validation and generalizability assessment. The authors also mention using subsets of the eICU Collaborative Research Database and the Adult (Census Income) Dataset for further validation of NCR.

Key Insights and Ideas

  • CV Techniques Translate to Tabular Data: A significant insight is that CV techniques, despite the fundamental differences between image and tabular data, can be effectively adapted to mitigate label noise in EHRs. This opens up a new avenue for improving ML models in healthcare.
  • NCR and Mix-up Shine: Neighbor Consistency Regularization (NCR) and Mix-up emerged as the most promising methods, consistently outperforming baselines and showing robustness even with substantial label corruption.
  • Combinations are Powerful: Combining Mix-up and NCR yielded the best overall performance, suggesting that synergistic effects between different noise-mitigation strategies can be highly beneficial.
  • NCR's Generalization Power: NCR not only improved performance on noisy data but also showed comparable or even superior results to standard cross-entropy loss on clean data, indicating a general regularization effect that enhances model robustness.
  • Warm-up Period for NCR is Key: For NCR, a warm-up period where the model is initially trained with only cross-entropy loss before introducing the NCR term significantly improved performance, suggesting a curriculum learning approach.

Key Results

  • Improved AUROC: The study demonstrated significant improvements in Area Under the Receiver Operating Characteristic Curve (AUROC) across various noise levels and test sets. For instance, NCR achieved up to a 3.7% increase in AUROC compared to the baseline NN model.
  • Robustness to Noise: Models trained with NCR exhibited lower standard deviations in performance across different noise ratios, indicating more consistent and reliable results.
  • Comparable to Clean Data Performance: NCR's performance was often comparable to, or even better than, standard cross-entropy on data with 0% added noise, highlighting its general regularization benefits.
  • Label Smoothing Underperforms: Label smoothing was found to be the least effective method, particularly when dealing with high levels of label noise or when the majority of labels were correct.
  • Practical Utility Demonstrated: The methods were validated on real-world COVID-19 diagnosis data, demonstrating their practical utility in a critical healthcare application.

Limitations and Future Work

  • Feature Noise vs. Label Noise: The study primarily focused on label noise. The authors acknowledge that errors can also persist within the features themselves (feature noise), which was not the main focus. Future work could explore methods to address this.
  • Missing Data: The study encountered missing data, which was handled using population median imputation. However, the authors note that the nature and reasons for missing data could hold valuable information for future analysis.
  • Feature Importance: Techniques like Mix-up can alter feature importance by smoothing decision boundaries. Further research into model explainability and how these techniques impact feature interpretation is suggested.
  • Limited Dataset Scope: While the COVID-19 datasets are valuable, they represent only a portion of the information available in EHR systems. Future research should explore diverse EHR data types and a wider range of noise mitigation techniques to assess their influence on model performance.
  • Binary Classification: The study focused on binary classification (COVID-19 positive/negative). The authors suggest that probabilities from models could be valuable for specific tasks, aligning with the continuous outputs of Mix-up and Label Smoothing.

Summary updated

Abstract

The analysis of extensive electronic health records (EHR) datasets often calls for automated solutions, with machine learning (ML) techniques, including deep learning (DL), taking a lead role. One common task involves categorizing EHR data into predefined groups. However, the vulnerability of EHRs to noise and errors stemming from data collection processes, as well as potential human labeling errors, poses a significant risk. This risk is particularly prominent during the training of DL models, where the possibility of overfitting to noisy labels can have serious repercussions in healthcare. Despite the well-documented existence of label noise in EHR data, few studies have tackled this challenge within the EHR domain. Our work addresses this gap by adapting computer vision (CV) algorithms to mitigate the impact of label noise in DL models trained on EHR data. Notably, it remains uncertain whether CV methods, when applied to the EHR domain, will prove effective, given the substantial divergence between the two domains. We present empirical evidence demonstrating that these methods, whether used individually or in combination, can substantially enhance model performance when applied to EHR data, especially in the presence of noisy/incorrect labels. We validate our methods and underscore their practical utility in real-world EHR data, specifically in the context of COVID-19 diagnosis. Our study highlights the effectiveness of CV methods in the EHR domain, making a valuable contribution to the advancement of healthcare analytics and research.
BibTeX
@inproceedings{yang2024addressing,
  author = {Jenny Yang and Hagen Triendl and Andrew A. S. Soltan and Mangal Prakash and David A. Clifton},
  title = {{Addressing label noise for electronic health records: insights from computer vision for tabular data}},
  booktitle = {BMC Medical Informatics Decis. Mak},
  year = {2024},
  url = {https://openreview.net/forum?id=Dficj7wtCt},
}
Figure