TL;DR
This paper explores the use of Large Language Models (LLMs) for anonymizing clinical text, a crucial step for enabling data sharing while protecting patient privacy. They introduce six novel evaluation metrics to fairly assess anonymization quality and compare LLM-based methods against traditional techniques, finding LLMs to be a promising alternative.
What Problem Does This Paper Address?
Clinical text, such as patient records and doctor's notes, contains highly sensitive personal information. Sharing this data is vital for research, policy development, and improving healthcare, but it's heavily restricted by privacy regulations like GDPR and HIPAA. Automated de-identification (anonymization) aims to remove or mask this sensitive information, but existing methods often fall short. They can be too complex, fail to anonymize effectively, or significantly degrade the utility of the data, making healthcare institutions hesitant to adopt them. This paper addresses the challenge of finding robust, effective, and trustworthy anonymization techniques, specifically investigating the potential of modern LLMs to overcome the limitations of previous approaches.
Methods
The authors propose six new evaluation metrics designed to address the unique challenges of generative anonymization, moving beyond traditional precision and recall. These metrics fall into two categories:
- Anonymization Sensitivity Metrics: These focus on how well sensitive entities are masked. They leverage the Levenshtein Distance (LD) to measure string similarity and include:
- Average Levenshtein Index of Dissimilarity (ALID): Measures the average maximum similarity between sensitive entities and text segments in the anonymized note.
- Levenshtein Recall (LR): Calculates the proportion of sensitive entities whose similarity to any text segment falls below a certain threshold.
- Levenshtein Recall for Direct Identifiers (LRDI) & Quasi-identifiers (LRQI): Specialized versions of LR that treat direct identifiers (like names) more strictly than quasi-identifiers (like dates).
- Clinical Information Retention Metrics: These assess how much of the original clinical meaning is preserved after anonymization. They utilize a BioBERT model pre-trained on ICD-10 codes:
- Jaccard Similarity Coefficient (JSC): Compares the binary representations of ICD-10 code category presence in the original and anonymized notes.
- Normalized Softmax Discounted Cumulative Gain (NSDCG): Compares the ranking of ICD-10 code logits between the original and anonymized notes, using a softmax discount for more nuanced weighting.
The paper compares seven anonymization strategies: two baselines (Microsoft Presidio and KNEO) and five LLM-based approaches (a fine-tuned ClinicalBERT, and prompt-based Phi-2 and Llama-3 models, including zero-shot and fine-tuned versions).
Datasets
The study uses the MIMIC III dataset, comprising 66,645 discharge summary notes. For training and validation, 70% of the data was used, with the remaining 30% reserved for testing. To simulate real-world anonymization challenges, fake sensitive entities were introduced into the dataset using the Faker library, categorized according to common anonymization tags (e.g., names, dates, locations).
Key Insights and Ideas
- LLMs are a viable alternative: The study strongly suggests that LLMs, particularly when fine-tuned, offer a promising path towards robust clinical text anonymization, potentially surpassing traditional methods.
- New metrics are crucial: Traditional metrics like precision and recall are insufficient for evaluating generative anonymization. The proposed LD-based and BioBERT-based metrics provide a more nuanced and fair assessment of both privacy preservation and data utility.
- Fine-tuning matters: While larger LLMs show promise, fine-tuning them on clinical data significantly improves their ability to anonymize accurately and retain clinical context, outperforming even larger zero-shot models.
- KNEO's limitations: The KNEO baseline, while good at string matching, leads to a substantial loss of clinical information, highlighting the trade-off between aggressive anonymization and data utility.
- Challenges in evaluation: Perfect recall is difficult to achieve due to inherent ambiguities in clinical notes (e.g., name initials appearing in non-sensitive words) and potential labeling inconsistencies in datasets.
Key Results
- Anonymization Sensitivity: ClinicalBERT and fine-tuned LLMs (Phi-2, Llama-3) generally performed best across sensitivity metrics like SMR and LR. KNEO was only the best by SMR, indicating it aggressively replaces tokens but might not be as nuanced. Presidio showed weaker performance in several sensitivity metrics.
- Clinical Information Retention: Fine-tuned generative models (Phi-2, Llama-3) achieved the highest scores in JSC and NSDCG, indicating better preservation of clinical concepts. ClinicalBERT also performed well. KNEO and Presidio showed significantly lower information retention.
- Parameter Size vs. Fine-tuning: Increasing LLM parameter count generally had a positive impact, but fine-tuning was essential for maximizing precision and clinical information retention, with even smaller fine-tuned models outperforming larger zero-shot models.
Limitations and Future Work
The authors acknowledge several limitations:
- LD-based metric similarities: LD-based metrics can sometimes identify strong similarities between unrelated text spans (e.g., "Tim" and "time"), potentially underestimating anonymization performance.
- LRQI and combined effects: LRQI evaluates quasi-identifiers individually, potentially overlooking their combined privacy risk.
- JSC binarization: The binarization step in JSC makes it insensitive to nuanced differences in logit distributions.
- Threshold dependency: LR, LRDI, LRQI, and JSC rely on thresholds that may require domain expertise and adjustment for different datasets.
- BioBERT limitations: Using a BioBERT model pre-trained on ICD-10 classification for information retention might be flawed with highly anonymized text, as the model wasn't specifically fine-tuned for anonymized data. This reliance could introduce biases.
Future work could focus on improving these metrics, exploring more robust ground truth models for evaluation, and further refining LLM-based anonymization techniques to balance privacy and utility more effectively.
Summary updated