TL;DR
This paper investigates whether de-identifying clinical text, a crucial step for privacy, harms its usefulness for downstream NLP tasks and human analysis. They found that de-identification, even with potential noise, does not significantly degrade the utility of clinical data for either machine learning models or human evaluators.
What Problem Does This Paper Address?
Electronic health records (EHRs) are a treasure trove of information for medical research and developing AI tools for healthcare. However, they contain sensitive Protected Health Information (PHI) that can identify patients, making their use and sharing difficult due to privacy concerns. De-identification aims to remove this PHI to enable broader data access. The challenge is that automatic de-identification methods, while improving, are not perfect. They can introduce "noise" into the data by either missing PHI or incorrectly removing non-sensitive information. This raises a critical question: does this noise introduced by de-identification degrade the quality and utility of the clinical text for its intended purposes, such as training NLP models or for human researchers? Lawyers also express concerns that de-identification might compromise data safety.
Methods
The researchers employed a multi-pronged approach to evaluate the impact of de-identification:
- Automatic De-identification: They utilized two versions of the HB Deid system for de-identification. The first uses Conditional Random Fields (CRF), and the second, more advanced version, uses a clinical BERT model fine-tuned for Named Entity Recognition (NER). This BERT model was pre-trained on general domain data and then adapted to the clinical domain using sensitive clinical data.
- Quantitative Evaluation (NLP Tasks): They trained and evaluated BERT models on three different clinical datasets (Gastro ICD-10 Corpus, Diagnosis Factuality Corpus, and Clinical Entity Corpus) using both de-identified and original (unaltered) data. The performance was measured using F1 scores on downstream tasks like ICD-10 block prediction, diagnosis factuality prediction, and Named Entity Recognition.
- Qualitative Evaluation (Human Assessors): Two types of qualitative studies were conducted.
- Study I: Two human evaluators (a coordinating officer and a chief physician) reviewed de-identified clinical data to assess its perceived utility and safety. They also participated in discussions to define what constitutes sensitive information.
- Study II: A computer scientist evaluated de-identified patient records from an emergency unit to assess the quality of the de-identification process itself.
Datasets
The study utilized several Swedish clinical datasets:
- Stockholm EPR Gastro ICD-10 Corpus: A large corpus of discharge summaries for gastrointestinal diseases, used for ICD-10 code prediction.
- Stockholm EPR Diagnosis Factuality Corpus: Clinical notes annotated with the factuality of diagnoses, used for predicting diagnosis factuality levels.
- Stockholm EPR Clinical Entity Corpus: Clinical text annotated with entities like Diagnosis, Drugs, Body parts, and Findings, used for Named Entity Recognition.
- Gastro Pseudo Clinical Dataset: A de-identified and pseudonymized subset of the Stockholm EPR Gastro ICD-10 Corpus, used for qualitative experiments.
- Emergency Unit Patient Records: 100 patient records from an emergency unit at Skåne University Hospital, used for a second qualitative study.
Key Insights and Ideas
- De-identification doesn't kill utility: The core finding is that de-identification, despite introducing noise, does not significantly harm the utility of clinical text for downstream NLP tasks. This is a crucial reassurance for researchers and developers.
- Human evaluators are surprisingly resilient: Human evaluators were less sensitive to the noise introduced by de-identification than might be expected. In fact, one evaluator found it difficult to distinguish between real and de-identified data, suggesting that pseudonymized records retain much of their utility for non-NLP research.
- Replacing with class names is preferred over surrogates: Human evaluators preferred replacing sensitive entities with their class names (e.g., "First Name") rather than arbitrary surrogate values. This approach, while potentially less "hidden," was seen as more transparent.
- Abbreviations are a persistent challenge: The study highlighted that abbreviations can be easily misclassified as organizations, posing a challenge for de-identification systems. The authors suggest using word lists to address this.
Key Results
- Quantitative: Models trained on pseudonymized data performed indistinguishably from models trained on unaltered data across three different NLP tasks. Wilcoxon rank-sum tests confirmed no statistically significant differences in F1 scores.
- Qualitative:
- Human evaluators found pseudonymized text not difficult to read.
- One evaluator struggled to distinguish between real and pseudonymized data.
- The de-identification system was generally effective, though some problematic cases (e.g., misidentified names/locations) were noted but confirmed as correctly tagged in the de-identified data.
- The system using class names for replacement was found to be easier to read than one using surrogates.
Limitations and Future Work
- Limitations:
- The study acknowledges that while de-identification doesn't harm downstream performance, the "noise" introduced could still be a concern for specific, highly sensitive analyses.
- The qualitative study involved a small number of evaluators, limiting generalizability.
- The authors note that the "HIPS" (Hiding In Plain Site) approach, where PHI is replaced by class names, makes it obvious that PHI was present, which might be a concern for some security perspectives.
- Future Work:
- The authors suggest that addressing abbreviations with specific word lists could improve de-identification.
- Further research could explore the trade-offs between different de-identification strategies and their impact on various downstream tasks and user groups.
Summary updated