Abstract
This study systematically investigates how errors in unstructured text data—such as spelling mistakes and missing whitespaces in clinical notes—affect the performance of downstream machine learning models. By employing a dual approach of corrupting clean data and cleaning real-world erroneous data using Large Language Models (LLMs), the authors identify a critical threshold: models remain robust to noise levels below 10%, but performance degrades significantly beyond this point. The research also highlights that traditional feature representations like TF-IDF can outperform neural embeddings in noisy clinical environments.
Introduction
In healthcare, unstructured text data like clinical progress notes are vital for predictive modeling but are often riddled with informal abbreviations, typos, and formatting errors. While the impact of data quality on structured data is well-understood, its effect on unstructured text remains under-explored. This paper addresses whether these textual errors materially hurt model performance and evaluates the utility of using LLMs for data cleaning.
Methodology
The researchers adopted a two-pronged experimental framework to isolate the effects of data quality on predictive tasks.
- Systematic Degradation: High-quality data (MIMIC-III) was synthetically corrupted with error rates ranging from 5% to 20% to observe the breakdown point of various models.
- Systematic Correction: Real-world "messy" data (Aged Care Homes dataset) was cleaned using the Mixtral 8x7B LLM to determine if automated correction improves predictive accuracy.
The study compared traditional feature extraction (TF-IDF) against neural embeddings (word2vec and Clinical BERT) across several classifiers, including Logistic Regression, Random Forests, and XGBoost.
Data Quality and Error Types
Clinical notes present unique challenges due to the variety of errors introduced during rapid documentation. The authors categorized these into several types, ranging from simple spelling mistakes to complex grammatical issues and domain-specific informal abbreviations.
The raw Aged Care Homes (ACH) dataset exhibited an average token-level error rate of approximately 7.5%, primarily consisting of the categories illustrated above.
Key Findings
The 10% Tipping Point
A primary discovery of this research is the "10% threshold." Machine learning models demonstrated remarkable resilience to minor textual noise. However, once the error rate reached or exceeded 10%, predictive performance (measured by ROC-AUC) dropped precipitously. This suggests that for many real-world datasets with low error rates (like the 7.5% found in the ACH data), intensive cleaning may yield diminishing returns.
Feature Representation Robustness
Surprisingly, traditional TF-IDF features consistently outperformed deep learning embeddings like word2vec and Clinical BERT across these specific medical tasks. Neural embeddings proved more sensitive to noise; for instance, word2vec often ignores out-of-vocabulary (OOV) words created by typos, thereby discarding potentially predictive information. In contrast, TF-IDF's reliance on individual word frequencies preserved more signal despite the noise.
LLMs as Data Cleaners
While the Mixtral 8x7B model was effective at identifying and correcting standard errors (achieving 63% detection accuracy), it struggled with domain-specific medical jargon. It frequently misidentified valid medical abbreviations (e.g., "mane", "obs", "RN") as errors, leading to "hallucinated" corrections that could potentially distort the data's meaning.
Conclusion
This study provides a practical guideline for data scientists working with clinical text: data cleaning becomes a priority only when error rates exceed 10%. Furthermore, it underscores the continued relevance of traditional NLP techniques like TF-IDF in specialized, noisy domains where deep learning models might struggle with out-of-vocabulary issues. Future work will focus on domain-adapted LLMs to better handle medical terminology and longer document contexts.
Summary updated