TL;DR
This paper explores generating privacy-preserving synthetic clinical notes using large language models (LLMs) by comparing different generation methods. They found that a "keyword-based" approach offers a strong balance between data utility (for tasks like ICD-9 coding) and privacy, outperforming simpler "one-shot" generation methods that risk exposing more sensitive information.
What Problem Does This Paper Address?
The increasing digitization of healthcare through Electronic Health Records (EHRs) offers immense potential for research and improved patient care. However, these records contain highly sensitive Protected Health Information (PHI), making data sharing for research a significant privacy challenge. While Natural Language Processing (NLP) techniques can help de-identify PHI, current methods often struggle to fully comply with privacy regulations like HIPAA, leading to a scarcity of accessible, high-quality datasets. This limits research and the development of AI models in healthcare. The paper addresses this by investigating how LLMs can generate synthetic clinical notes that are both privacy-preserving and retain enough utility for downstream tasks, thus facilitating safer data sharing.
Methods
The researchers experimented with three LLMs: GPT-3.5, GPT-4, and Mistral 7B. They employed three text generation approaches:
- One-shot: Directly using original (de-identified) clinical notes as prompts for the LLMs.
- Normalized One-shot: Similar to one-shot, but the input text was first normalized by removing numbers and symbols, keeping only alphabetic words.
- Keyword: This method involved extracting key phrases from normalized notes using unsupervised keyword extraction algorithms (KP-miner and YAKE), then using these keywords to generate synthetic notes. This approach aims to retain essential information while stripping away potentially identifying details.
Privacy was assessed by analyzing the occurrence and co-occurrence of PHI in the generated notes. Utility was evaluated using an ICD-9 coding classification task. Text quality was measured using ROUGE and Cosine Similarity metrics to compare synthetic notes with original ones.
Datasets
The study utilized a subset of the MIMIC III database, specifically 9,817 discharge summaries. This dataset was filtered to focus on single primary diagnoses from eight major disease categories. The researchers also used re-identified versions of this dataset to compare privacy and utility outcomes.
Key Insights and Ideas
- Keyword generation is the privacy champion: This method significantly reduced PHI occurrence and co-occurrence compared to the more direct "one-shot" approaches.
- "One-shot" is a privacy risk: Directly feeding original notes into LLMs, even after de-identification, led to higher rates of PHI exposure, especially for geographic location and dates.
- Re-identified data can be more useful: Surprisingly, synthetic notes generated from re-identified data (where placeholders were filled with dummy PHI) often showed better utility (e.g., in ICD-9 coding) than those from de-identified data. This suggests a potential avenue for enhancing synthetic data utility without compromising privacy.
- Geographic location is a persistent challenge: This PHI category was the most prevalent across models and generation methods, even in keyword-based approaches, highlighting its inherent difficulty in anonymization.
- LLMs can sometimes "over-remember": Unique identifiers, which are typically rare in general LLM training data, were sometimes fully extracted and reappeared in synthetic notes, posing a privacy risk.
Key Results
- Privacy: The Keyword generation method showed the lowest PHI occurrence, with geographic location being the primary remaining risk. One-shot generation had the highest PHI exposure.
- Utility (ICD-9 Coding): The Normalized One-shot generation achieved the highest classification accuracy, with some models even surpassing the performance of real, de-identified benchmark data. Keyword generation showed slightly lower performance but offered better privacy.
- Text Similarity: Mistral 7B in the Normalized One-shot generation achieved the highest ROUGE scores, indicating good text quality. Cosine Similarity results showed that Keyword and One-shot approaches generally maintained better semantic similarity than Normalized One-shot, especially when using re-identified data.
Limitations and Future Work
The authors acknowledge several limitations:
- Comparison with existing NER models: Direct comparison with established Named Entity Recognition (NER) models for PHI detection was not feasible due to the novel evaluation approach used in this study.
- Real-world scenario nuances: The study used a simplified dataset (single coding scenarios) and may not fully capture the complexity of real-world EHR data, which often involves multi-coding and nuanced language.
- LLM variability: The performance of LLMs can vary significantly, and smaller models sometimes outperform larger ones, making standardized evaluation crucial.
- Rapidly evolving field: The fast pace of LLM development means findings may quickly become outdated, necessitating continuous evaluation.
Future work could involve exploring alternative coding classifications, incorporating more complex multi-coding scenarios, and investigating emerging technologies for even more robust privacy and utility in synthetic EHR generation.
Summary updated