DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4

Zhengliang Liu, Xiaowei Yu, Lu Zhang, Zihao Wu, Chao Cao, Haixing Dai, Lin Zhao, Wei Liu, Dinggang Shen, Quanzheng Li, Tianming Liu, Dajiang Zhu, Xiang Li

Abstract

The digitization of healthcare has facilitated the sharing and re-using of medical data but has also raised concerns about confidentiality and privacy. HIPAA (Health Insurance Portability and Accountability Act) mandates removing re-identifying information before the dissemination of medical records. Thus, effective and efficient solutions for de-identifying medical data, especially those in free-text forms, are highly needed. While various computer-assisted de-identification methods, including both rule-based and learning-based, have been developed and used in prior practice, such solutions still lack generalizability or need to be fine-tuned according to different scenarios, significantly imposing restrictions in wider use. The advancement of large language models (LLM), such as ChatGPT and GPT-4, have shown great potential in processing text data in the medical domain with zero-shot in-context learning, especially in the task of privacy protection, as these models can identify confidential information by their powerful named entity recognition (NER) capability. In this work, we developed a novel GPT4-enabled de-identification framework ("DeID-GPT") to automatically identify and remove the identifying information. Compared to existing commonly used medical text data de-identification methods, our developed DeID-GPT showed the highest accuracy and remarkable reliability in masking private information from the unstructured medical text while preserving the original structure and meaning of the text. This study is one of the earliest to utilize ChatGPT and GPT-4 for medical text data processing and de-identification, which provides insights for further research and solution development on the use of LLMs such as ChatGPT/GPT-4 in healthcare.

The Evolution of De-identification

In the era of big data, sharing clinical notes is essential for medical research, yet the Health Insurance Portability and Accountability Act (HIPAA) strictly mandates the removal of 18 categories of personal identifiers. Historically, de-identification has been a major bottleneck. Manual redaction is prohibitively expensive and slow, while rule-based systems are often too rigid to handle the linguistic nuances of "free-text" notes. Traditional machine learning models, such as ClinicalBERT, require massive amounts of labeled data and intensive fine-tuning for every new hospital or dataset.

The historical progression of HIPAA de-identification methods from manual expert annotation to modern LLM-based approaches.

The authors sought a "universal" solution that could handle diverse medical texts with high reliability and zero task-specific training, leading to the development of the LLM-based DeID-GPT framework.

The DeID-GPT Framework

DeID-GPT moves away from the traditional "train-then-test" paradigm toward a "prompt-and-process" approach. The methodology consists of two primary stages:

  1. Semantic Similarity Mapping: To ensure the model understands the specific requirements of a dataset, the framework uses a semantic similarity voting approach to map standard HIPAA identifiers to the specific categories used in a given benchmark.
  2. Explicit Prompt Engineering: The core of the system is an optimized prompt template. The researchers found that "Implicit" prompts were insufficient. Instead, they utilized "Explicit" prompts that broke instructions into a task statement, specific commands, and detailed rules with examples of what to redact.
The DeID-GPT workflow: integrating HIPAA identifiers into an explicit prompt to transform an original clinical report into a de-identified (De-ID) version.

The framework was tested primarily on GPT-4 and ChatGPT (GPT-3.5) in a zero-shot setting, meaning the models were given no prior examples of the task before being asked to perform it.

Benchmarking and Performance

The study utilized the 2014 i2b2/UTHealth de-identification challenge dataset, containing 1,304 free-form clinical notes. The quantitative results place GPT-4 at the top of the leaderboard for medical de-identification:

  • GPT-4 (Explicit Prompt): Achieved an entity-wise accuracy of 0.99, the highest reported in the study.
  • ChatGPT (GPT-3.5): Reached 0.929 with explicit prompts, showing a significant gap between the two model generations.
  • ClinicalBERT (Fine-tuned): Reached 0.974. While strong, it still fell short of GPT-4's zero-shot performance.

Beyond raw accuracy, GPT-4 demonstrated remarkable robustness to prompt variations and successfully removed sensitive data while keeping the clinical meaning and grammatical structure of the notes intact—a vital requirement for downstream research.

Limitations and Future Directions

Despite high accuracy, several hurdles remain for real-world adoption. Sending sensitive patient data to cloud-based APIs like OpenAI's presents regulatory challenges. Future work should focus on deploying open-source LLMs (such as LLaMA) locally within hospital firewalls. Additionally, while LLMs are powerful, the costs and latency of API calls for millions of records must be considered. The authors propose using GPT-4 to generate high-quality synthetic data to fine-tune smaller, local models, combining the intelligence of frontier LLMs with the security of on-site deployment.

Summary updated

Abstract

The digitization of healthcare has facilitated the sharing and re-using of medical data but has also raised concerns about confidentiality and privacy. HIPAA (Health Insurance Portability and Accountability Act) mandates removing re-identifying information before the dissemination of medical records. Thus, effective and efficient solutions for de-identifying medical data, especially those in free-text forms, are highly needed. While various computer-assisted de-identification methods, including both rule-based and learning-based, have been developed and used in prior practice, such solutions still lack generalizability or need to be fine-tuned according to different scenarios, significantly imposing restrictions in wider use. The advancement of large language models (LLM), such as ChatGPT and GPT-4, have shown great potential in processing text data in the medical domain with zero-shot in-context learning, especially in the task of privacy protection, as these models can identify confidential information by their powerful named entity recognition (NER) capability. In this work, we developed a novel GPT4-enabled de-identification framework ("DeID-GPT") to automatically identify and remove the identifying information. Compared to existing commonly used medical text data de-identification methods, our developed DeID-GPT showed the highest accuracy and remarkable reliability in masking private information from the unstructured medical text while preserving the original structure and meaning of the text. This study is one of the earliest to utilize ChatGPT and GPT-4 for medical text data processing and de-identification, which provides insights for further research and solution development on the use of LLMs such as ChatGPT/GPT-4 in healthcare. Codes and benchmarking data information are available at https://github.com/yhydhx/ChatGPT-API.
BibTeX
@misc{liu2023deid,
  author = {Zhengliang Liu and Xiaowei Yu and Lu Zhang and Zihao Wu and Chao Cao and Haixing Dai and Lin Zhao and Wei Liu and Dinggang Shen and Quanzheng Li and Tianming Liu and Dajiang Zhu and Xiang Li},
  title = {{DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4}},
  year = {2023},
  eprint = {2303.11032},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.11032},
}
Figure