Generalizing Clinical De-identification Models by Privacy-safe Data Augmentation using GPT-4

Woojin Kim, Sungeun Hahm, Jaejin LeeEMNLP 2024 · OpenReview

Abstract

Clinical de-identification (de-ID) is essential for maintaining patient confidentiality while enabling secondary research on medical records. However, state-of-the-art models often suffer from a "generalization gap," performing poorly when applied to datasets from different institutions or time periods. This study introduces a privacy-safe framework that utilizes GPT-4 for data augmentation through one-shot and zero-shot prompting. By redacting Protected Health Information (PHI) before processing and subsequently re-populating generated text with surrogate data, our approach ensures privacy while significantly enhancing model robustness. Experimental results on i2b2 benchmarks demonstrate substantial improvements in F1 scores, particularly at the entity level.

Introduction

The removal of PHI from clinical notes is a critical bottleneck in healthcare data management. While deep learning models like BioBERT achieve high accuracy on internal datasets, their performance typically plummets in cross-dataset scenarios. This lack of generalization is driven by data scarcity—due to strict HIPAA regulations—and institutional variance in templates, shorthand, and labeling standards. Existing rule-based augmentation methods often fail to capture the complex linguistic diversity of real-world clinical narratives. We propose a pipeline that leverages the creative capacity of Large Language Models (LLMs) like GPT-4 to diversify training context without risking PHI leakage.

Methodology

Our "Privacy-safe Data Augmentation" pipeline consists of three distinct stages designed to maintain strict data security while maximizing linguistic variety:

  1. PHI-Scrubbing: Before any data is transmitted to the GPT-4 API, real PHI in the training set is replaced with generic placeholders (e.g., "John" becomes [PATIENT], "Nov 12" becomes [DATE]). This ensures that sensitive information never leaves the local environment.
  2. GPT-4 Generation:
    • One-shot: The model is provided with one scrubbed clinical note and prompted to generate a linguistically distinct but structurally similar report, maintaining the original placeholders.
  3. Zero-shot: GPT-4 generates synthetic clinical notes from scratch based solely on provided annotation guidelines.
  4. PHI Augmentation: The generic placeholders in the synthetic text are filled with "surrogate" PHI—realistic but fake names, addresses, and dates—sourced from public databases or regular expressions.
The proposed privacy-safe data augmentation pipeline (A) and the cross-dataset testing framework (B).

The resulting augmented datasets are used to fine-tune BERT-based models (BioBERT and Bio-ClinicalBERT) for the Named Entity Recognition (NER) task of de-identification.

Experimental Setup and Datasets

The framework was evaluated using two gold-standard benchmarks: i2b2 2006 (discharge summaries) and i2b2 2014 (diverse clinical notes with complex PHI). To ensure compatibility, PHI labels were consolidated into five core categories: Name, Location, Date, ID, and Contact.

Key Results

The evaluation focused on cross-dataset performance (e.g., training on i2b2 2006 plus augmented data and testing on i2b2 2014).

  • Significant Generalization Gains: For the BioBERT model, the Entity-level micro F1 score improved from 66.79% to 82.29% when moving from the 2006 to the 2014 dataset—a nearly 16-point jump.
  • Superiority over Rule-based Methods: The GPT-4 one-shot method consistently outperformed PHICON, a prior state-of-the-art rule-based augmentation technique.
  • Zero-shot Viability: Models trained on purely synthetic (Zero-shot) data outperformed cross-dataset baselines, suggesting that LLMs possess sufficient latent medical knowledge to simulate realistic clinical narratives without exposure to real patient data.
Performance comparison across PHI categories for Base, Zero-shot, and One-shot augmentation methods, showing consistent improvements in the 'Overall' metric.

Discussion and Limitations

A key insight of this work is that GPT-4's ability to diversify the context surrounding PHI is more valuable for model generalization than simply swapping entities. The most significant gains were observed in Entity-level F1 scores, indicating that the model learned more precise boundaries for PHI detection.

However, some limitations remain. In zero-shot settings, performance in "Location" and "Name" categories was slightly lower than the baseline. This is likely because GPT-4 generates general names and locations, whereas clinical records often contain highly specific formats like internal hospital wing names. Additionally, performance tended to saturate at an augmentation level of $\alpha=4$ or $5$, beyond which the model may begin to overfit to the LLM's specific writing style.

Conclusion

This study demonstrates that privacy-safe data augmentation using GPT-4 can effectively bridge the generalization gap in clinical de-identification. By focusing on contextual diversity and maintaining a strict scrubbing protocol, we provide a viable path for developing robust models that can be deployed across diverse medical institutions without compromising patient privacy.

Summary updated

Abstract

De-identification (de-ID) refers to removing the association between a set of identifying data and the data subject. In clinical data management, the de-ID of Protected Health Information (PHI) is critical for patient confidentiality. However, state-of-the-art de-ID models show poor generalization on a new dataset. This is due to the difficulty of retaining training corpora. Additionally, labeling standards and the formats of patient records vary across different institutions. Our study addresses these issues by exploiting GPT-4 for data augmentation through one-shot and zero-shot prompts. Our approach effectively circumvents the problem of PHI leakage, ensuring privacy by redacting PHI before processing. To evaluate the effectiveness of our proposal, we conduct cross-dataset testing. The experimental result demonstrates significant improvements across three types of F1 scores.
BibTeX
@inproceedings{kim2024generalizing,
  author = {Woojin Kim and Sungeun Hahm and Jaejin Lee},
  title = {{Generalizing Clinical De-identification Models by Privacy-safe Data Augmentation using GPT-4}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=4STs0plO0K},
}
Figure