Abstract
Glaucoma is a leading cause of irreversible blindness worldwide, with early intervention often being crucial. Research into the underpinnings of glaucoma often relies on electronic health records (EHRs) to identify patients with glaucoma and their subtypes. However, current methods for identifying glaucoma patients from EHRs are often inaccurate or infeasible at scale, relying on International Classification of Diseases (ICD) codes or manual chart reviews. To address this limitation, we introduce (1) OphthaBERT, a powerful general clinical ophthalmology language model trained on over 2 million diverse clinical notes, and (2) a fine-tuned variant of OphthaBERT that automatically extracts binary and subtype glaucoma diagnoses from clinical notes. The base OphthaBERT model is a robust encoder, outperforming state-of-the-art clinical encoders in masked token prediction on out-of-distribution ophthalmology clinical notes and binary glaucoma classification with limited data. We report significant binary classification performance improvements in low-data regimes (p < 0.001, Bonferroni corrected). OphthaBERT is also able to achieve superior classification performance for both binary and subtype diagnosis, outperforming even fine-tuned large decoder-only language models at a fraction of the computational cost. We demonstrate a 0.23-point increase in macro-F1 for subtype diagnosis over ICD codes and strong binary classification performance when externally validated at Wilmer Eye Institute. OphthaBERT provides an interpretable, equitable framework for general ophthalmology language modeling and automated glaucoma diagnosis.
Introduction
Glaucoma research at scale is significantly hindered by the unreliability of structured data in Electronic Health Records (EHRs). International Classification of Diseases (ICD) codes, while commonly used for administrative purposes, often fail to reflect clinical reality, with accuracy rates sometimes as low as 50%. The most valuable clinical information—such as cup-to-disc ratios and specific disease progression—is typically buried in unstructured clinical notes.
Manually reviewing these notes is labor-intensive and impractical for large-scale studies. Furthermore, general-purpose medical AI models often lack the specialized vocabulary required for ophthalmology. OphthaBERT was developed to bridge this gap, providing an accurate, efficient, and equitable tool for extracting glaucoma diagnoses from clinical narratives.
Methods
OphthaBERT was developed using a two-stage training approach designed to instill domain-specific expertise into a lightweight architecture.
- Domain-Specific Pretraining: Starting with BioClinicalBERT, the model underwent "continued masked language modeling" (MLM) on 2.1 million ophthalmology-specific notes. This process allowed the model to learn specialized jargon and clinical context.
- Supervised Fine-Tuning: The model was then equipped with two parallel classification heads: one for binary diagnosis (presence of glaucoma) and another for subtype classification (identifying one of seven specific glaucoma types).
Unlike large decoder-only models (e.g., GPT-4), OphthaBERT is an encoder-based model. This architecture makes it significantly more efficient for classification tasks while maintaining high performance.
Datasets and Evaluation
The model was trained and validated using a diverse set of clinical data:
- Pretraining Corpus: Over 2.1 million de-identified notes from Massachusetts Eye and Ear (MEE) spanning 2016 to 2024.
- Internal Validation: 3,003 notes manually labeled by clinicians to establish a ground truth.
- External Validation: Notes from Oregon Health & Science University (OHSU) and the Wilmer Eye Institute at Johns Hopkins University (JHU) were used to test the model's generalizability across different hospital systems.
To ensure equity, the researchers intentionally sampled notes from White, Black, and Asian patient groups, confirming that the model performs consistently across different racial demographics.
Results and Insights
OphthaBERT demonstrated substantial improvements over existing methods. In masked token prediction, it achieved a 26% absolute increase in top-1 accuracy over BioClinicalBERT. For binary glaucoma classification, the model reached an ROC-AUC of 0.97.
Key insights from the study include:
- Specialization Over Scale: Despite having only 2% of the parameters of massive models like MedAlpaca-7B, OphthaBERT achieved superior subtype classification.
- Superiority to ICD Codes: The model outperformed ICD-based diagnosis by 0.23 points in macro-F1 score for subtype identification.
- Interpretability: Using Integrated Gradients, the team verified that the model focuses on clinically significant terms like "cup," "ratio," and "suspect," rather than administrative noise.
Discussion and Limitations
While OphthaBERT is highly effective, it is currently limited by the 512-token context window of the BERT architecture. Extremely long clinical notes may be truncated, potentially losing information if diagnostic details are buried deep in the text. Future iterations may explore "Longformer" architectures to accommodate longer narratives.
OphthaBERT represents a significant step forward in automated clinical phenotyping, offering a scalable and interpretable solution for ophthalmology research and practice.
Summary updated