Abstract: This study explored the usability of prompt generation on named entity recognition (NER) tasks and the performance in different settings of the prompt. The prompt generation by GPT-J models was utilized to directly test the gold standard as well as to generate the seed and further fed to the RoBERTa model with the spaCy package. In the direct test, a lower ratio of negative examples with higher numbers of examples in prompt achieved the best results with a F1 score of 0.72. The performance revealed consistency, 0.92-0.97 in the F1 score, in all settings after training with the RoBERTa model. The study highlighted the importance of seed quality rather than quantity in feeding NER models. This research reports on an efficient and accurate way to mine clinical notes for periodontal diagnoses, allowing researchers to easily and quickly build a NER model with the prompt generation approach.
Introduction
In 2018, the classification system for periodontitis (gum disease) was updated to a more complex framework requiring the documentation of "Stage" (severity), "Grade" (risk of progression), and "Extent" (distribution). Because this system is relatively new, many dentists record these diagnoses within unstructured free-text clinical notes rather than structured data fields. For researchers, mining these notes is traditionally a labor-intensive process requiring thousands of manually labeled examples for Named Entity Recognition (NER) models. Furthermore, using cloud-based large language models (LLMs) poses significant HIPAA privacy risks when dealing with Protected Health Information (PHI).
Methodology
This research proposes a two-stage hybrid NLP pipeline to bridge the gap between unstructured text and structured clinical data:
- Seed Generation (The LLM Phase): The researchers used GPT-J-6B, an open-source model deployed locally to ensure data privacy. Using "few-shot" prompting, GPT-J was tasked with extracting Stage, Grade, and Extent from clinical notes. The team experimented with different prompt configurations, including varying sample sizes (15, 25, 50 examples) and the ratio of "negative" examples (sentences containing no diagnosis).
- NER Training (The RoBERTa Phase): The labels generated by GPT-J (the "seeds") were used to fine-tune a RoBERTa-base model using the spaCy library. While GPT-J is powerful, it is computationally expensive; RoBERTa acts as a faster, more stable, and specialized extractor for production use.
Datasets
The study utilized 8,125 clinical notes from Electronic Dental Records (EDR) at the University of Texas Health Science Center at Houston. A "Gold Standard" of 60 notes was manually labeled by an experienced dentist to serve as the ground truth for evaluation.
Key Results
The study found that GPT-J could effectively act as a "silver standard" annotator. When GPT-J was used directly to predict the gold standard, it achieved a peak F1 score of 0.72. Performance in this direct testing phase was highest when using a lower ratio of negative examples and a larger sample size of 50 examples in the prompt.
Once the RoBERTa model was fine-tuned on the GPT-J-generated seeds, performance improved dramatically. Across various prompt settings, the RoBERTa model achieved F1 scores between 0.92 and 0.97. This high level of consistency suggests that the specialized model could generalize effectively from the LLM's initial annotations.
Discussion and Insights
- Quality Over Quantity: The research highlighted that the quality of the seeds was more important than the volume. Even when the number of extracted sentences decreased in certain prompt settings, the final RoBERTa model's performance remained high.
- The "Combined" Prompt Advantage: Prompting the model to extract all three entities (Stage/Grade/Extent) simultaneously worked significantly better than asking for them individually, which often led to hallucinations or missing data.
- Privacy-First Architecture: By deploying GPT-J and RoBERTa locally, the researchers demonstrated that high-end NLP can be performed within a secure clinical environment without sending sensitive patient data to external APIs.
Conclusion
This study demonstrates an efficient and accurate way to mine clinical notes for complex diagnoses. By using LLMs to bootstrap smaller, specialized models, researchers can build high-performing NER tools with minimal manual labeling, facilitating easier large-scale clinical research while maintaining data privacy.
Summary updated