Using Synthetic Health Care Data to Leverage Large Language Models for Named Entity Recognition: Development and Validation Study

Hendrik Šuvalov, Mihkel Lepson, Veronika Kukk, Maria Malk, Neeme Ilves, Hele-Andra Kuulmets, Raivo KoldeJournal of Medical Internet Research 2025 · DOI

TL;DR

This paper tackles the challenge of building accurate Named Entity Recognition (NER) models for low-resource languages like Estonian, particularly in the sensitive domain of healthcare. They propose a novel approach using Large Language Models (LLMs) to generate synthetic healthcare data and then annotate it, bypassing the need for expensive and privacy-risky human annotation. This method successfully trains a functional NER model, demonstrating a promising path for developing AI tools in under-resourced linguistic and medical contexts.

What Problem Does This Paper Address?

Extracting valuable information from electronic health records (EHRs) is crucial for medical research, decision support, and data mining. Named Entity Recognition (NER) is a key technology for this, identifying entities like diseases, medications, and procedures. However, developing robust NER models for low-resource languages, such as Estonian, is severely hampered by the scarcity of annotated data. This scarcity is exacerbated in healthcare due to the highly sensitive and personal nature of patient data, making it difficult to collect and share for annotation. Existing methods often rely on large amounts of human-annotated data, which is expensive, time-consuming, and raises privacy concerns. This paper aims to overcome these limitations by creating a pipeline that leverages LLMs to generate synthetic data and perform annotations, thereby preserving patient privacy and enabling NER model development for languages with limited resources.

Methods

The paper outlines a three-step pipeline:

  1. Synthetic Data Generation: A locally trained GPT-2 model is used to generate synthetic Estonian healthcare records. This is done to avoid sending sensitive real patient data to external LLM providers. The training data for this GPT-2 model consisted of approximately 10 million texts from Estonian discharge reports. To ensure the synthetic data wasn't too close to the original, they employed a longest common subsequence (LCS) algorithm to filter out near-duplicates.
  2. LLM-based Annotation: The generated synthetic data is then fed into more powerful LLMs (GPT-3.5-Turbo and GPT-4) via API calls. The LLMs are prompted to annotate specific medical entities. To optimize this process, the researchers experimented with four different prompt strategies: zero-shot, Estonian zero-shot, definitions-based, and few-shot. The output from the LLMs is parsed into a structured format (JSON) for downstream use.
  3. Downstream NER Model Training: The LLM-annotated synthetic data is used to fine-tune a transformer-based NER model. Specifically, they chose the XLM-RoBERTa base model, which is pre-trained on Estonian and other similar languages, making it suitable for this task. This fine-tuned model is then evaluated on real-world medical texts.

Datasets

  • Training Data for GPT-2: Approximately 10 million texts from 200,000 Estonians, collected between 2010 and 2020, comprising national registry of discharge reports covering primary and secondary care contacts. This data was used to train the synthetic data generator.
  • Synthetic Health Care Data: Generated by the GPT-2 model. Initially, around 4100 documents (2700 medical history, 1400 procedures) were deemed suitable after LCS filtering.
  • Real-World Medical Texts: Used for evaluating the final trained NER model. This included 150 patient medical histories and 150 patient procedure texts.
  • Human Annotated Data: A subset of the real-world medical texts was annotated by two medical experts to serve as ground truth for evaluating the LLM annotations and the final model performance.

Key Insights and Ideas

  • Synthetic Data as a Privacy-Preserving Solution: The core innovation is using LLMs to generate and annotate synthetic data, effectively sidestepping the privacy concerns associated with using real patient records for training. This is particularly critical for sensitive domains like healthcare.
  • LLMs as Powerful Annotators: LLMs like GPT-3.5-Turbo and GPT-4 can perform NER annotations with reasonable accuracy, even for a low-resource language like Estonian, when provided with appropriate prompts.
  • Prompt Engineering is Key: The choice of prompt significantly impacts the quality of LLM annotations. The "few-shot" prompt, which provides examples, and the "definitions" prompt, which clarifies entity types, generally outperformed simpler prompts.
  • English Prompts Outperform Estonian Prompts: Despite the target language being Estonian, prompts written in English yielded better results. This highlights the inherent bias in LLMs towards English due to its overrepresentation in training data.
  • Diminishing Returns with More Data: The study found that beyond a certain point (around 2000 synthetic texts for drugs), adding more synthetic training data did not significantly improve the NER model's performance, suggesting a saturation point or potential for overfitting.
  • Interannotator Agreement Highlights Subjectivity: The relatively moderate interannotator agreement among human experts for certain entity types (especially procedures) underscores that even "ground truth" can be subjective, setting a realistic benchmark for automated systems.

Key Results

  • Top-Performing NER Model: The best-performing NER model achieved an F₁-score of 0.69 for drug extraction and 0.38 for procedure extraction on real-world medical data.
  • LLM Performance Comparison: GPT-4 generally outperformed GPT-3.5-Turbo and the local Llammas model in terms of F₁-score across most entity types. However, GPT-3.5-Turbo showed higher precision for procedures and smoking.
  • Prompt Performance: The "few-shot" prompt yielded the highest F₁-scores for drugs, procedures, and smoking when used with GPT-3.5-Turbo. The "definitions" prompt was best for disease extraction.
  • Data Volume Impact: For drug extraction, the model's performance peaked around 2000 synthetic texts. For procedures, the peak F₁-score was achieved with 1000 synthetic texts.

Limitations and Future Work

  • Limitations:
    • The F₁-score for procedures (0.38) is still relatively low, indicating challenges in accurately extracting this entity type.
  • The performance of the local Llammas model was limited, partly due to its smaller size and issues with parsing responses.
  • The study acknowledges that while their results are promising for low-resource languages, they do not surpass state-of-the-art transformer models trained on abundant human-annotated data for high-resource languages.
  • Future Work:
    • Refining the synthetic data generation process by using newer, more advanced LLM architectures.
  • Exploring more specialized models for clinical texts (e.g., Me-LLaMA, GatorTron) to improve annotation accuracy.
  • Further prompt engineering to enhance LLM annotation quality.
  • Developing more detailed annotation guidelines for both human and LLM annotators.
  • Expanding the method's applicability to other domains and languages.

Summary updated

Abstract

This paper demonstrates a successful approach to leveraging LLMs for training NER models using synthetic data, effectively preserving patient privacy. By avoiding reliance on human-annotated data, our method shows promise in developing models for low-resource languages, such as Estonian. Future work will focus on refining the synthetic data generation and expanding the method's applicability to other domains and languages.
BibTeX
@article{suvalov2025using,
  author = {Hendrik Šuvalov and Mihkel Lepson and Veronika Kukk and Maria Malk and Neeme Ilves and Hele-Andra Kuulmets and Raivo Kolde},
  title = {{Using Synthetic Health Care Data to Leverage Large Language Models for Named Entity Recognition: Development and Validation Study}},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/66279},
}
Figure