Self-Supervised Detection of Contextual Synonyms in a Multi-Class Setting: Phenotype Annotation Use Case

Jingqing Zhang, Luis Bolanos, Tong Li, Ashwani Tanwar, Guilherme Freire, Xian Yang, Julia Ive, Vibhor Gupta, Yike Guo

Abstract: Jingqing Zhang, Luis Bolanos Trujillo, Tong Li, Ashwani Tanwar, Guilherme Freire, Xian Yang, Julia Ive, Vibhor Gupta, Yike Guo. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

Introduction

Extracting "phenotypes"—observable physical or biochemical characteristics—from Electronic Health Records (EHRs) is critical for disease diagnosis and precision medicine. However, this task faces three significant hurdles: extreme multi-class sparsity (over 15,000 concepts in the Human Phenotype Ontology), high contextual variation in clinical language (e.g., "low pressure" vs. "Hypotension"), and the prohibitive cost of expert-labeled data.

This paper introduces a self-supervised framework that leverages ontology-aware training objectives and data augmentation to detect contextual synonyms across thousands of classes, achieving state-of-the-art results with minimal labeled data.

Methodology

The proposed approach builds on ClinicalBERT using a "silver standard" generation process. Simple keyword matching identifies exact mentions of HPO terms in raw text, which are then used as training signals to teach the model to recognize the surrounding context.

Model architecture and training objectives.

Key Technical Innovations

  • Three-Pronged Training Objective:
    • Objective 1 (Binary): Distinguishes phenotype tokens from non-phenotype tokens.
  • Objective 2 (Frequent Classes): Uses standard cross-entropy loss for common HPO concepts.
  • Objective 3 (Rare Classes): Minimizes the Euclidean distance between a token’s contextual embedding and a pre-trained Knowledge Graph (KG) embedding of the HPO concept. This allows the model to generalize to rare classes based on their semantic position in the ontology.
  • Data Augmentation: Employs lexical pivoting (back-translation) and a Seq2Seq model to generate synthetic clinical context for rare HPO terms.
  • KG Embeddings: HPO concept embeddings are trained using relational and semantic losses to ensure similar medical concepts have similar vector representations.

Experimental Evaluation

The model was evaluated across diverse clinical and scientific corpora, including MIMIC-III (ICU records), COVID-I & II (radiology reports), and PubMed (scientific abstracts).

Performance across datasets and data percentages.

Key Results

  • Unsupervised SOTA: In the self-supervised setting, the model outperformed previous state-of-the-art methods (like NCR and NCBO) by up to 4.5 F1 points and 4.0 Recall points on clinical text.
  • Data Efficiency: When fine-tuned with human labels, the model reached the performance level of a fully-supervised BioBERT using only 20% of the training data.
  • Recall Improvement: The method specifically targets higher recall by identifying contextual synonyms that traditional keyword searches miss.
  • Downstream Impact: Phenotype annotations generated by the model significantly improved performance in tasks like predicting ICU length-of-stay and mortality.

Discussion and Conclusion

The model demonstrates exceptional performance on clinical notes but shows less improvement on PubMed abstracts, where scientific literature follows stricter naming conventions that favor shallow matching. A limitation noted is the difficulty in capturing highly nuanced radiographic concepts in specific domains like COVID-19.

Overall, the framework proves that anchoring rare classes to their positions in a Knowledge Graph allows models to recognize phenotypes without requiring massive labeled datasets. Future work will explore more sophisticated text generation for data augmentation and application to other large-scale medical ontologies.

Summary updated

BibTeX
@inproceedings{zhang2021self,
  author = {Jingqing Zhang and Luis Bolanos and Tong Li and Ashwani Tanwar and Guilherme Freire and Xian Yang and Julia Ive and Vibhor Gupta and Yike Guo},
  title = {{Self-Supervised Detection of Contextual Synonyms in a Multi-Class Setting: Phenotype Annotation Use Case}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=madFJmoGoc},
}
Figure