TL;DR
This paper presents a method for automatically extracting periodontitis diagnoses from unstructured clinical notes using a combination of regular expressions (RE) and the RoBERTa transformer model. By leveraging advanced NLP techniques, the study demonstrates a significant improvement in identifying these crucial diagnoses, addressing a gap in structured EHR data.
What Problem Does This Paper Address?
Electronic Health Records (EHRs) are a treasure trove of patient information, but much of it is locked away in free-text clinical notes, making it difficult to analyze. This is particularly true in dentistry, where diagnoses, especially for conditions like periodontitis, are often not recorded in a structured format. This lack of structured diagnostic information poses a challenge for quality patient care, continuity of care, and even insurance claims, as future providers may not have a clear picture of a patient's history. Manually extracting this information is time-consuming and prone to errors. The paper aims to bridge this gap by developing an automated system to extract periodontitis diagnoses from these unstructured notes.
Methods
The study employs a two-pronged approach: regular expressions (RE) for initial data extraction and a RoBERTa transformer model for named entity recognition (NER).
- Regular Expressions (RE): Two levels of RE complexity were explored:
- Simple RE: This approach looked for specific labels (diagnosis, extent, stage, grade) in a predefined order within the notes.
-
Advanced RE: This version was more flexible, not requiring labels to be in a specific order and allowing for partial matches. These RE methods were used to generate initial training data for the NER model.
-
Named Entity Recognition (NER) with RoBERTa: The SpaCy package was used for tokenization, and the RoBERTa base model was chosen for its effectiveness in general-domain tasks. RoBERTa, a powerful transformer model, was fine-tuned on the data extracted by the RE methods to identify and classify periodontitis-related diagnostic entities. A "combined" approach was also tested, leveraging the advanced RE results and supplementing them with the simple RE where necessary.
Datasets
The study utilized clinical notes from the University of Texas Health Science Center at Houston (UTHealth) dental clinics.
- Initial Dataset: Data was extracted from EHRs between January 1, 2021, and December 31, 2021, focusing on patients with complete periodontal charting. This resulted in 8,125 clinical notes.
- Gold Standard: A subset of 60 clinical notes was manually labeled by an experienced dentist to serve as the ground truth for evaluating the models. These notes were excluded from the training data.
- Training Data: A total of 8,065 clinical notes were used for generating training data after filtering.
Key Insights and Ideas
- RE Complexity Matters: The study clearly shows that as the complexity of regular expressions increases, their ability to accurately extract relevant information (measured by F1 score) significantly improves, moving from around 0.3-0.4 to nearly 0.9.
- NLP Augments RE: While RE can be effective, it has limitations. The RoBERTa NER model, trained on RE-extracted data, demonstrated superior performance, achieving much higher accuracy in identifying diagnoses. This highlights the power of combining rule-based extraction with sophisticated machine learning models.
- Bridging the Diagnosis Gap: The research directly addresses the critical issue of missing structured diagnoses in clinical notes, offering a practical solution that can enhance patient care and data analysis.
- Generalizability Potential: The authors suggest that this approach could be extended beyond periodontitis to other dental diseases and even to different medical fields, indicating broad applicability.
Key Results
- RE Performance: The simple RE method achieved F1 scores of 0.3-0.4, while the advanced RE method reached scores around 0.9.
- NER Model Performance:
- The simple RE-trained NER model achieved evaluation metrics between 0.84-0.92.
- The advanced and combined RE-trained NER models demonstrated excellent predictions, with scores ranging from 0.95-0.99.
- Specific Metrics: Table IV shows that the combined approach achieved a perfect 1.0 F1 score for Stage and near-perfect scores for Extent (0.96-0.97) and Grade (0.98). Precision and recall were consistently high across all labels for the advanced and combined models.
Limitations and Future Work
- Limitations:
- False Negatives: A common issue of high false negatives was observed, attributed to the high diversity and informal language in clinical notes, making it difficult to capture all rules.
- Typos and Informal Terms: The NER model sometimes captured terms with typos or informal variations, necessitating post-processing for standardization.
- Complexity of Notes: Some diagnoses were written in less common formats or included explanations that made direct extraction challenging.
- Model Imperfection: The authors acknowledge that no model is perfect, and even the advanced RE and NER models had occasional misses.
- Future Work:
- Testing on Diverse Datasets: The model should be tested on clinical notes from other institutions and across different medical fields.
- Incorporating Rare Terminologies: Enhancing the model's ability to handle rare medical terms is crucial.
- Exploring Advanced Models: Integrating larger language models like GPT could further optimize the pipeline for even better results.
- Extending to Other Diseases: Applying the methodology to extract diagnoses for other dental or medical conditions.
Summary updated