Abstract
Clinical de-identification (de-ID) is essential for maintaining patient confidentiality while enabling secondary research on medical records. However, state-of-the-art models often suffer from a "generalization gap," performing poorly when applied to datasets from different institutions or time periods. This study introduces a privacy-safe framework that utilizes GPT-4 for data augmentation through one-shot and zero-shot prompting. By redacting Protected Health Information (PHI) before processing and subsequently re-populating generated text with surrogate data, our approach ensures privacy while significantly enhancing model robustness. Experimental results on i2b2 benchmarks demonstrate substantial improvements in F1 scores, particularly at the entity level.
Introduction
The removal of PHI from clinical notes is a critical bottleneck in healthcare data management. While deep learning models like BioBERT achieve high accuracy on internal datasets, their performance typically plummets in cross-dataset scenarios. This lack of generalization is driven by data scarcity—due to strict HIPAA regulations—and institutional variance in templates, shorthand, and labeling standards. Existing rule-based augmentation methods often fail to capture the complex linguistic diversity of real-world clinical narratives. We propose a pipeline that leverages the creative capacity of Large Language Models (LLMs) like GPT-4 to diversify training context without risking PHI leakage.
Methodology
Our "Privacy-safe Data Augmentation" pipeline consists of three distinct stages designed to maintain strict data security while maximizing linguistic variety:
- PHI-Scrubbing: Before any data is transmitted to the GPT-4 API, real PHI in the training set is replaced with generic placeholders (e.g., "John" becomes
[PATIENT], "Nov 12" becomes[DATE]). This ensures that sensitive information never leaves the local environment. - GPT-4 Generation:
- One-shot: The model is provided with one scrubbed clinical note and prompted to generate a linguistically distinct but structurally similar report, maintaining the original placeholders.
- Zero-shot: GPT-4 generates synthetic clinical notes from scratch based solely on provided annotation guidelines.
- PHI Augmentation: The generic placeholders in the synthetic text are filled with "surrogate" PHI—realistic but fake names, addresses, and dates—sourced from public databases or regular expressions.
The resulting augmented datasets are used to fine-tune BERT-based models (BioBERT and Bio-ClinicalBERT) for the Named Entity Recognition (NER) task of de-identification.
Experimental Setup and Datasets
The framework was evaluated using two gold-standard benchmarks: i2b2 2006 (discharge summaries) and i2b2 2014 (diverse clinical notes with complex PHI). To ensure compatibility, PHI labels were consolidated into five core categories: Name, Location, Date, ID, and Contact.
Key Results
The evaluation focused on cross-dataset performance (e.g., training on i2b2 2006 plus augmented data and testing on i2b2 2014).
- Significant Generalization Gains: For the BioBERT model, the Entity-level micro F1 score improved from 66.79% to 82.29% when moving from the 2006 to the 2014 dataset—a nearly 16-point jump.
- Superiority over Rule-based Methods: The GPT-4 one-shot method consistently outperformed PHICON, a prior state-of-the-art rule-based augmentation technique.
- Zero-shot Viability: Models trained on purely synthetic (Zero-shot) data outperformed cross-dataset baselines, suggesting that LLMs possess sufficient latent medical knowledge to simulate realistic clinical narratives without exposure to real patient data.
Discussion and Limitations
A key insight of this work is that GPT-4's ability to diversify the context surrounding PHI is more valuable for model generalization than simply swapping entities. The most significant gains were observed in Entity-level F1 scores, indicating that the model learned more precise boundaries for PHI detection.
However, some limitations remain. In zero-shot settings, performance in "Location" and "Name" categories was slightly lower than the baseline. This is likely because GPT-4 generates general names and locations, whereas clinical records often contain highly specific formats like internal hospital wing names. Additionally, performance tended to saturate at an augmentation level of $\alpha=4$ or $5$, beyond which the model may begin to overfit to the LLM's specific writing style.
Conclusion
This study demonstrates that privacy-safe data augmentation using GPT-4 can effectively bridge the generalization gap in clinical de-identification. By focusing on contextual diversity and maintaining a strict scrubbing protocol, we provide a viable path for developing robust models that can be deployed across diverse medical institutions without compromising patient privacy.
Summary updated