Self-Supervised Detection of Contextual Synonyms in a Multi-Class Setting: Phenotype Annotation Use Case

Jingqing Zhang, Luis Bolanos, Tong Li, Ashwani Tanwar, Guilherme Freire, Xian Yang, Julia Ive, Vibhor Gupta, Yike Guo

Abstract

Contextualised word embeddings is a powerful tool to detect contextual synonyms. However, most of the current state-of-the-art (SOTA) deep learning concept extraction methods remain supervised and underexploit the potential of the context. In this paper, we propose a self-supervised pre-training approach which is able to detect contextual synonyms of concepts being training on the data created by shallow matching. We apply our methodology in the sparse multi-class setting (over 15,000 concepts) to extract phenotype information from electronic health records. We further investigate data augmentation techniques to address the problem of the class sparsity. Our approach achieves a new SOTA for the unsupervised phenotype concept annotation on clinical text on F1 and Recall outperforming the previous SOTA with a gain of up to 4.5 and 4.0 absolute points, respectively. After fine-tuning with as little as 20% of the labelled data, we also outperform BioBERT and ClinicalBERT. The extrinsic evaluation on three ICU benchmarks also shows the benefit of using the phenotypes annotated by our model as features.

Introduction

Extracting structured medical concepts from unstructured clinical text, such as Electronic Health Records (EHRs), is a fundamental challenge in clinical informatics. While supervised deep learning models have shown promise, they are constrained by the need for extensive, expert-labeled datasets. Conversely, unsupervised methods often rely on shallow keyword matching, which fails to capture "contextual synonyms"—the varied, informal ways clinicians describe medical conditions (e.g., "BP of 79/48" for Hypotension). This paper addresses these challenges by proposing a self-supervised approach for large-scale phenotype extraction, focusing on the Human Phenotype Ontology (HPO) which contains over 15,000 concepts.

Methodology

The proposed framework leverages ClinicalBERT to handle the extreme multi-class nature of the HPO through a three-pronged objective strategy. First, a binary classifier identifies HPO-relevant text spans. Second, a multi-class classifier handles frequent HPO concepts using standard cross-entropy. Third, for rare concepts, the model uses a Euclidean distance metric to align contextualized token embeddings with pre-trained Knowledge Graph (KG) embeddings of HPO concepts. These KG embeddings are trained separately to capture both hierarchical relationships (relational loss) and semantic definitions (semantic loss).

Overview of the self-supervised training framework, illustrating the three-pronged objective and the integration of HPO knowledge graph embeddings.

To further mitigate class sparsity, the authors utilize advanced data augmentation techniques, including English-French-English back-translation, mapping abnormal lab values to phenotypes (e.g., "Hgb 5" to Anemia), and synthetic context generation for rare concepts using a Seq2Seq model.

Results and Discussion

The model was evaluated on several benchmarks, including MIMIC-III, COVID-19 radiology reports (COVID-I and COVID-II), and PubMed abstracts. In an unsupervised setting, the model achieved state-of-the-art (SOTA) performance, outperforming previous methods like NCBO and NCR by up to 5 absolute points in F1 score on clinical datasets.

A key finding is the efficiency of the model in low-resource scenarios. As shown in Figure 2, when fine-tuned on just 20% of labeled data using uncertainty sampling, the model outperforms a ClinicalBERT baseline fine-tuned on 100% of the data.

Performance comparison (Exact Match F1 Score) across MIMIC, COVID-II, and PubMed datasets, highlighting the effectiveness of uncertainty sampling and fine-tuning.

Extrinsic evaluation confirmed the utility of the extracted phenotypes for downstream tasks. Incorporating these phenotypes as features in ICU predictive models led to significant improvements: Length-of-stay Kappa increased from 0.380 to 0.430, and In-hospital Mortality AUROC increased from 0.826 to 0.845.

Conclusion

This research demonstrates that a self-supervised approach, combining contextualized embeddings with knowledge graph representations, can effectively bridge the gap between unstructured clinical text and formal ontologies. While the model shows some domain sensitivity—performing better on clinical notes than on scientific literature—it sets a new benchmark for phenotype annotation and highlights the value of contextual synonyms in clinical data science. Future work may focus on refining knowledge graph embeddings to better distinguish between closely related hierarchical concepts.

Summary updated

Abstract

Contextualised word embeddings is a powerful tool to detect contextual synonyms. However, most of the current state-of-the-art (SOTA) deep learning concept extraction methods remain supervised and underexploit the potential of the context. In this paper, we propose a self-supervised pre-training approach which is able to detect contextual synonyms of concepts being training on the data created by shallow matching. We apply our methodology in the sparse multi-class setting (over 15,000 concepts) to extract phenotype information from electronic health records. We further investigate data augmentation techniques to address the problem of the class sparsity. Our approach achieves a new SOTA for the unsupervised phenotype concept annotation on clinical text on F1 and Recall outperforming the previous SOTA with a gain of up to 4.5 and 4.0 absolute points, respectively. After fine-tuning with as little as 20\% of the labelled data, we also outperform BioBERT and ClinicalBERT. The extrinsic evaluation on three ICU benchmarks also shows the benefit of using the phenotypes annotated by our model as features.
BibTeX
@inproceedings{zhang2021self,
  author = {Jingqing Zhang and Luis Bolanos and Tong Li and Ashwani Tanwar and Guilherme Freire and Xian Yang and Julia Ive and Vibhor Gupta and Yike Guo},
  title = {{Self-Supervised Detection of Contextual Synonyms in a Multi-Class Setting: Phenotype Annotation Use Case}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=madFJmoGoc},
}
Figure