Two Directions for Clinical Data Generation with Large Language Models: Data-to-Label and Label-to-Data

Rumeng Li, Xun Wang, hong yu

Abstract

Large language models (LLMs) can generate natural language texts for various domains and tasks, but their potential for clinical text mining, a domain with scarce, sensitive, and imbalanced medical data, is under-explored. We investigate whether LLMs can augment clinical data for detecting Alzheimer's Disease (AD)-related signs and symptoms from electronic health records (EHRs), a challenging task that requires high expertise. We create a novel pragmatic taxonomy for AD sign and symptom progression based on expert knowledge and generated three datasets: (1) a gold dataset annotated by human experts on longitudinal EHRs of AD patients; (2) a silver dataset created by the data-to-label method, which labels sentences from a public EHR collection with AD-related signs and symptoms; and (3) a bronze dataset created by the label-to-data method which generates sentences with AD-related signs and symptoms based on the label definition. We train a system to detect AD-related signs and symptoms from EHRs. We find that the silver and bronze datasets improves the system performance, outperforming the system using only the gold dataset. This shows that LLMs can generate synthetic clinical data for a complex task by incorporating expert knowledge, and our label-to-data method can produce datasets that are free of sensitive information, while maintaining acceptable quality.

Introduction: The Clinical Data Bottleneck

Clinical text mining—extracting insights from doctor's notes and electronic health records (EHRs)—is a high-stakes field plagued by three major hurdles: scarcity, sensitivity, and imbalance. Medical data is expensive to annotate because it requires specialized expertise. It is also highly sensitive; strict privacy laws like HIPAA make sharing real patient data difficult. Finally, clinical data is often imbalanced, with rare but critical symptoms appearing far less frequently than common ones. This paper investigates whether LLMs can bypass these issues by acting as either expert annotators or creative data generators.

Methodology: Silver and Bronze Data

The authors proposed a novel 9-category taxonomy for Alzheimer’s Disease (AD) progression (e.g., Cognitive impairment, Neuropsychiatric symptoms, Coping strategies) and explored two distinct directions for data augmentation:

  1. Data-to-Label (Silver Data): Researchers used de-identified clinical notes from the public MIMIC-III database and employed Llama 65B to label them according to their taxonomy. To ensure quality, they used a "chain-of-thought" style check where the model had to explain its reasoning.
  2. Label-to-Data (Bronze Data): They used GPT-4 to generate entirely synthetic clinical sentences based only on the label definitions and expert guidelines. This method is particularly promising because the resulting data contains zero real patient information, making it inherently privacy-preserving.

To test these datasets, they trained an ensemble of three pre-trained language models (BERT, RoBERTa, and ClinicalBERT) and evaluated them on a "Gold" test set annotated by human medical professionals.

Dataset Characteristics

The study utilized three tiers of data:

  • Gold Dataset: 16,194 sentences from 5,112 longitudinal EHR notes, annotated by two medical professionals.
  • Silver Dataset: 16,069 sentences from MIMIC-III, labeled by Llama 65B.
  • Bronze Dataset: 16,047 entirely synthetic sentences generated by GPT-4.

One key insight was that the synthetic "Bronze" data was much more balanced across categories than the real-world "Gold" data. This helped the final model perform significantly better on minority classes—symptoms that rarely appear in actual medical records but are crucial for early diagnosis.

Key Results

The addition of LLM-generated data led to clear performance gains across the board. In binary classification (detecting if a sentence was AD-related or not), the system reached an accuracy of 0.94, representing a 4.44% improvement over using Gold data alone.

Binary classification performance for AD-related sign and symptom detection.

Multi-class classification tasks, which assigned sentences to one of the nine categories, saw an overall accuracy boost of 5.88%. The most dramatic improvements were seen in rare categories. For example, performance on "Coping strategy" detection improved by a staggering 31.82% when using the combined Bronze and Silver datasets.

Multi-class classification performance across different dataset combinations.

Discussion: Hallucination as a Feature

In the "Label-to-Data" (Bronze) approach, the LLM’s tendency to "hallucinate" is actually beneficial. It creates diverse, realistic clinical scenarios that don't correspond to any real person, effectively bypassing privacy concerns while providing high-quality training material. However, human experts found that while the Bronze data had about 85% annotation accuracy, the Silver data (LLM-labeled real data) was only around 55%. The LLMs struggled with "over-inference" on real data—reading too much into a sentence—but were much more accurate when generating the context themselves.

Conclusion

This research demonstrates that LLMs can generate synthetic clinical data for complex tasks by incorporating expert knowledge. The label-to-data method, in particular, produces datasets that are free of sensitive information while maintaining acceptable quality, offering a promising path forward for privacy-preserving clinical AI.

Summary updated

Abstract

Large language models (LLMs) can generate natural language texts for various domains and tasks, but their potential for clinical text mining, a domain with scarce, sensitive, and imbalanced medical data, is under-explored. We investigate whether LLMs can augment clinical data for detecting Alzheimer's Disease (AD)-related signs and symptoms from electronic health records (EHRs), a challenging task that requires high expertise. We create a novel pragmatic taxonomy for AD sign and symptom progression based on expert knowledge and generated three datasets: (1) a gold dataset annotated by human experts on longitudinal EHRs of AD patients; (2) a silver dataset created by the data-to-label method, which labels sentences from a public EHR collection with AD-related signs and symptoms; and (3) a bronze dataset created by the label-to-data method which generates sentences with AD-related signs and symptoms based on the label definition. We train a system to detect AD-related signs and symptoms from EHRs. We find that the silver and bronze datasets improves the system performance, outperforming the system using only the gold dataset. This shows that LLMs can generate synthetic clinical data for a complex task by incorporating expert knowledge, and our label-to-data method can produce datasets that are free of sensitive information, while maintaining acceptable quality.
BibTeX
@inproceedings{li2023two,
  author = {Rumeng Li and Xun Wang and hong yu},
  title = {{Two Directions for Clinical Data Generation with Large Language Models: Data-to-Label and Label-to-Data}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=Yzi6LM20E2},
}
Figure