Abstract
Contextualised word embeddings is a powerful tool to detect contextual synonyms. However, most of the current state-of-the-art (SOTA) deep learning concept extraction methods remain supervised and underexploit the potential of the context. In this paper, we propose a self-supervised pre-training approach which is able to detect contextual synonyms of concepts being training on the data created by shallow matching. We apply our methodology in the sparse multi-class setting (over 15,000 concepts) to extract phenotype information from electronic health records. We further investigate data augmentation techniques to address the problem of the class sparsity. Our approach achieves a new SOTA for the unsupervised phenotype concept annotation on clinical text on F1 and Recall outperforming the previous SOTA with a gain of up to 4.5 and 4.0 absolute points, respectively. After fine-tuning with as little as 20% of the labelled data, we also outperform BioBERT and ClinicalBERT. The extrinsic evaluation on three ICU benchmarks also shows the benefit of using the phenotypes annotated by our model as features.
Introduction
Extracting structured medical concepts from unstructured clinical text, such as Electronic Health Records (EHRs), is a fundamental challenge in clinical informatics. While supervised deep learning models have shown promise, they are constrained by the need for extensive, expert-labeled datasets. Conversely, unsupervised methods often rely on shallow keyword matching, which fails to capture "contextual synonyms"—the varied, informal ways clinicians describe medical conditions (e.g., "BP of 79/48" for Hypotension). This paper addresses these challenges by proposing a self-supervised approach for large-scale phenotype extraction, focusing on the Human Phenotype Ontology (HPO) which contains over 15,000 concepts.
Methodology
The proposed framework leverages ClinicalBERT to handle the extreme multi-class nature of the HPO through a three-pronged objective strategy. First, a binary classifier identifies HPO-relevant text spans. Second, a multi-class classifier handles frequent HPO concepts using standard cross-entropy. Third, for rare concepts, the model uses a Euclidean distance metric to align contextualized token embeddings with pre-trained Knowledge Graph (KG) embeddings of HPO concepts. These KG embeddings are trained separately to capture both hierarchical relationships (relational loss) and semantic definitions (semantic loss).
To further mitigate class sparsity, the authors utilize advanced data augmentation techniques, including English-French-English back-translation, mapping abnormal lab values to phenotypes (e.g., "Hgb 5" to Anemia), and synthetic context generation for rare concepts using a Seq2Seq model.
Results and Discussion
The model was evaluated on several benchmarks, including MIMIC-III, COVID-19 radiology reports (COVID-I and COVID-II), and PubMed abstracts. In an unsupervised setting, the model achieved state-of-the-art (SOTA) performance, outperforming previous methods like NCBO and NCR by up to 5 absolute points in F1 score on clinical datasets.
A key finding is the efficiency of the model in low-resource scenarios. As shown in Figure 2, when fine-tuned on just 20% of labeled data using uncertainty sampling, the model outperforms a ClinicalBERT baseline fine-tuned on 100% of the data.
Extrinsic evaluation confirmed the utility of the extracted phenotypes for downstream tasks. Incorporating these phenotypes as features in ICU predictive models led to significant improvements: Length-of-stay Kappa increased from 0.380 to 0.430, and In-hospital Mortality AUROC increased from 0.826 to 0.845.
Conclusion
This research demonstrates that a self-supervised approach, combining contextualized embeddings with knowledge graph representations, can effectively bridge the gap between unstructured clinical text and formal ontologies. While the model shows some domain sensitivity—performing better on clinical notes than on scientific literature—it sets a new benchmark for phenotype annotation and highlights the value of contextual synonyms in clinical data science. Future work may focus on refining knowledge graph embeddings to better distinguish between closely related hierarchical concepts.
Summary updated