Abstract: Jingqing Zhang, Luis Bolanos Trujillo, Tong Li, Ashwani Tanwar, Guilherme Freire, Xian Yang, Julia Ive, Vibhor Gupta, Yike Guo. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.
Introduction
Extracting "phenotypes"—observable physical or biochemical characteristics—from Electronic Health Records (EHRs) is critical for disease diagnosis and precision medicine. However, this task faces three significant hurdles: extreme multi-class sparsity (over 15,000 concepts in the Human Phenotype Ontology), high contextual variation in clinical language (e.g., "low pressure" vs. "Hypotension"), and the prohibitive cost of expert-labeled data.
This paper introduces a self-supervised framework that leverages ontology-aware training objectives and data augmentation to detect contextual synonyms across thousands of classes, achieving state-of-the-art results with minimal labeled data.
Methodology
The proposed approach builds on ClinicalBERT using a "silver standard" generation process. Simple keyword matching identifies exact mentions of HPO terms in raw text, which are then used as training signals to teach the model to recognize the surrounding context.
Key Technical Innovations
- Three-Pronged Training Objective:
- Objective 1 (Binary): Distinguishes phenotype tokens from non-phenotype tokens.
- Objective 2 (Frequent Classes): Uses standard cross-entropy loss for common HPO concepts.
- Objective 3 (Rare Classes): Minimizes the Euclidean distance between a token’s contextual embedding and a pre-trained Knowledge Graph (KG) embedding of the HPO concept. This allows the model to generalize to rare classes based on their semantic position in the ontology.
- Data Augmentation: Employs lexical pivoting (back-translation) and a Seq2Seq model to generate synthetic clinical context for rare HPO terms.
- KG Embeddings: HPO concept embeddings are trained using relational and semantic losses to ensure similar medical concepts have similar vector representations.
Experimental Evaluation
The model was evaluated across diverse clinical and scientific corpora, including MIMIC-III (ICU records), COVID-I & II (radiology reports), and PubMed (scientific abstracts).
Key Results
- Unsupervised SOTA: In the self-supervised setting, the model outperformed previous state-of-the-art methods (like NCR and NCBO) by up to 4.5 F1 points and 4.0 Recall points on clinical text.
- Data Efficiency: When fine-tuned with human labels, the model reached the performance level of a fully-supervised BioBERT using only 20% of the training data.
- Recall Improvement: The method specifically targets higher recall by identifying contextual synonyms that traditional keyword searches miss.
- Downstream Impact: Phenotype annotations generated by the model significantly improved performance in tasks like predicting ICU length-of-stay and mortality.
Discussion and Conclusion
The model demonstrates exceptional performance on clinical notes but shows less improvement on PubMed abstracts, where scientific literature follows stricter naming conventions that favor shallow matching. A limitation noted is the difficulty in capturing highly nuanced radiographic concepts in specific domains like COVID-19.
Overall, the framework proves that anchoring rare classes to their positions in a Knowledge Graph allows models to recognize phenotypes without requiring massive labeled datasets. Future work will explore more sophisticated text generation for data augmentation and application to other large-scale medical ontologies.
Summary updated