TL;DR
This study investigates how textual data quality impacts machine learning model performance in healthcare. They found that models are surprisingly tolerant to minor errors (<10%), but performance degrades significantly with higher error rates (≥10%), emphasizing the need for data quality assessment and correction.
What Problem Does This Paper Address?
In real-world applications, especially in healthcare, textual data (like patient progress notes) is often messy, containing errors such as misspellings, grammatical mistakes, and informal language. This data quality issue is critical because incorrect predictions from machine learning models in healthcare can have severe consequences. While the impact of structured data quality on ML is well-studied, the effect of unstructured textual data quality, particularly in sensitive domains like healthcare, remains underexplored. This paper aims to quantify this impact, determine if models are robust to errors, and assess the value of investing in data quality improvement.
Methods
The researchers developed a rudimentary error rate metric by splitting text into tokens and identifying errors at that level, focusing on misspellings and missing whitespace. They used the Mixtral large language model (LLM) to both introduce synthetic errors into a high-quality dataset (MIMIC-III) and to correct errors in a real-world, lower-quality dataset (Australian Aged Care Homes - AACHs). They then evaluated the performance of various feature representations (TF-IDF and embeddings like word2vec, ClinicalBERT, BioClinical ModernBERT) and machine learning models (logistic regression, SVC, random forests, XGBoost, MLP) on these datasets with varying error levels. The primary evaluation metric was Area Under the Receiver Operating Curve (ROC-AUC).
Datasets
- MIMIC-III: A publicly available, deidentified critical care database from Beth Israel Deaconess Medical Center. This dataset was used as the high-quality baseline and had synthetic errors introduced.
- AACH Dataset: An anonymized, operational dataset from Australian Aged Care Homes. This dataset represented a real-world, lower-quality scenario and was used for error correction experiments.
Key Insights and Ideas
- LLMs as Data Quality Tools: Mixtral proved capable of both introducing controlled errors for experimentation and correcting errors in real-world datasets, demonstrating LLMs' potential in data quality management.
- Error Tolerance Threshold: Machine learning models show a surprising tolerance for textual data errors up to around 10%. Beyond this threshold, performance degradation becomes significant.
- Feature Representation Matters: Traditional TF-IDF features often outperformed deep learning embeddings (like word2vec and BERT variants) for the specific medical predictive tasks studied, suggesting that for certain clinical tasks, explicit keyword matching can be more robust to noise than contextual embeddings.
- Model Choice is Task-Dependent: Different machine learning models performed better on different datasets and tasks, highlighting that there's no one-size-fits-all solution.
Key Results
- Error Impact: For the MIMIC dataset, as error rates increased from 0% to 20% (in 5% increments), model performance (ROC-AUC) generally declined, with a noticeable drop occurring at error rates of 10% and above.
- AACH Dataset Quality: The AACH dataset had an average error rate of approximately 7.5%, which did not lead to a significant performance drop in the models used.
- Mixtral's Correction Efficacy: Mixtral successfully corrected about 54% of the errors in the AACH dataset, reducing its error rate from 7.5% to 3.4%.
- TF-IDF vs. Embeddings: In mortality prediction on MIMIC, TF-IDF features consistently yielded higher ROC-AUC scores compared to word2vec and ClinicalBERT embeddings across various error rates.
- Model Performance: Logistic regression often performed best with TF-IDF features for mortality prediction, while tree-based ensemble methods (like XGBoost) showed better performance on the AACH dataset.
Limitations and Future Work
- Rudimentary Error Metric: The error quantification metric was basic, focusing primarily on misspellings and missing whitespace, and did not fully account for complex errors like grammatical mistakes or informal abbreviations.
- Limited LLM Comparison: While Mixtral was used extensively, the study only performed a preliminary comparison with a few other open-source LLMs for error correction.
- Model and Feature Set: The study tested a limited selection of feature representations and machine learning models.
- Future Work: The authors suggest a more comprehensive evaluation of various LLMs for data correction, a deeper analysis of different error types (especially grammatical errors), and exploring a wider range of models and hyperparameter tuning. They also note that future research could explore methods for improving model tolerance to errors without relying on LLM-based correction.
Summary updated