Abstract
Electronic Health Records (EHR) store valuable patient-staff interaction data. Recent advancements in proprietary online large language models (LLMs) have shown promising capabilities in analyzing EHR notes. However, transmitting patient information through external APIs to LLMs like ChatGPT introduces privacy risks, necessitating alternative approaches that conform to hospital practices. To address privacy concerns, we propose generating synthetic documents based on a reward mechanism-trained model from real documents without leaking sensitive information but keeping relevant clinical knowledge. These synthetic documents may be annotated by large proprietary models or existing public ones, and used to train small specialized models that can run on constrained medical infrastructure. We validate our approach through a proof-of-concept scenario using Mimic-III, assessing the effectiveness of the generated documents through several downstream tasks: a series of ICD-9 multi-label classifications of varying complexity and a synthetic Named Entity Recognition (NER) task. The results demonstrate that synthetic documents preserve privacy and improve performance when real annotated data are sparse.
Introduction
The integration of Large Language Models (LLMs) into clinical workflows offers transformative potential for medical Natural Language Processing (NLP). However, the highly sensitive nature of Electronic Health Records (EHRs) presents a significant barrier. Regulations such as HIPAA and GDPR often prohibit the transmission of patient data to external APIs for analysis or annotation. While hospitals require specialized, small-scale models capable of running on local, resource-constrained infrastructure, training these models necessitates high-quality annotated data—a resource that is difficult to share and expensive to anonymize manually.
This paper introduces a "privacy-by-design" framework for generating synthetic clinical notes that retain medical utility without leaking sensitive patient information. By combining keyword-guided generation with a private reward mechanism using Direct Preference Optimization (DPO), we demonstrate that small models trained on these synthetic documents can match or exceed the performance of models trained on limited real-world data.
Methodology
The proposed pipeline bridges the gap between public model capabilities and private data security by splitting the process into a "Public Area" and a "Private Area."
Reward-based Generation Pipeline
- Knowledge Extraction: Non-sensitive clinical keywords (medical concepts) are extracted from real EHRs using QuickUMLS. These keywords serve as the "skeleton" for synthetic document generation.
- Seed Fine-Tuning: A base model (Mistral-7B-Instruct) is fine-tuned on a minimal, manually pseudo-anonymized "seed" dataset of 150–235 documents.
- Iterative Generation & Scoring: The generator produces multiple candidate documents based on the extracted keywords. These candidates are evaluated within the hospital's private infrastructure using SEMSCORE, a semantic similarity metric that compares synthetic text to original real text.
- Alignment via DPO: The generator is refined using Direct Preference Optimization (DPO), learning to prefer synthetic documents with high SEMSCOREs that mimic clinical logic without memorizing sensitive content.
- Downstream Training: High-quality synthetic documents are used to train a smaller "student" model (DeBERTa-v3-base) for specific tasks.
Experimental Setup
The framework was validated using the MIMIC-III dataset, specifically focusing on "History of Present Illness" (HPI) sections. Downstream tasks included multi-label ICD-9 classification (top 20, 50, 100, and 400 codes) and a Named Entity Recognition (NER) task. For NER, GPT-4 was used to annotate both real and synthetic documents to create a comparative benchmark.
Results and Discussion
The effectiveness of the reward mechanism is evidenced by the strong correlation between the SEMSCORE metric and downstream task performance across various levels of classification complexity.
Performance Scaling and Utility
Synthetic data proved particularly effective in scenarios where real annotated data was sparse. In the complex class-400 ICD-9 task, the synthetic-trained model achieved an F1 score of 31.0, significantly outperforming the model trained on the original small gold set (F1: 4.9). For NER, the synthetic-trained model reached an F1 of 61.7, comparable to oversampled real data (61.6).
Key Insights
- Privacy through Abstraction: Prompting with UMLS keywords forces the model to "hallucinate" narratives around medical facts rather than memorizing patient details.
- Consistency: Synthetic models exhibited lower standard deviations in performance compared to gold data models, suggesting a more stable training signal.
- The Private Evaluator Loop: This architecture allows public models to improve via private data without sensitive information leaving the hospital; only numerical scores are exchanged.
Limitations and Future Work
While promising, the approach has limitations. There remains a slight risk that rare sequences of keywords could be linked back to unique cases; future work will explore adding "noise" to keyword sequences. Additionally, while SEMSCORE is a useful proxy, it may not capture document-level coherence perfectly. Future research will expand testing to more complex reasoning tasks, such as medical question-answering, and investigate ways to reduce the computational cost of generating large synthetic datasets.
Summary updated