UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the Unified Medical Language System Metathesaurus

George Michalopoulos, Yuanxin Wang, Hussam Kaka, Helen H. Chen, Alexander Wong

Abstract

Contextual word embedding models, such as BioBERT and Bio_ClinicalBERT, have achieved state-of-the-art results in biomedical natural language processing tasks by focusing their pre-training process on domain-specific corpora. However, such models do not take into consideration expert domain knowledge. In this work, we introduced UmlsBERT, a contextual embedding model that integrates domain knowledge during the pre-training process via a novel knowledge augmentation strategy. More specifically, the augmentation on UmlsBERT with the Unified Medical Language System (UMLS) Metathesaurus was performed in two ways: i) connecting words that have the same underlying 'concept' in UMLS, and ii) leveraging semantic group knowledge in UMLS to create clinically meaningful input embeddings. By applying these two strategies, UmlsBERT can encode clinical domain knowledge into word embeddings and outperform existing domain-specific models on common named-entity recognition (NER) and clinical natural language inference clinical NLP tasks.

Introduction

While general-purpose language models like BERT are powerful, they often struggle with the specialized vocabulary of medicine. Previous attempts to adapt these models, such as BioBERT and Bio_ClinicalBERT, involved further pre-training on massive collections of unstructured biomedical papers and clinical notes. However, these models miss out on decades of human-curated, structured expert knowledge found in medical ontologies. For example, a standard model might not inherently know that "pulmonary" and "lungs" refer to the same concept. UmlsBERT bridges this gap by injecting the Unified Medical Language System (UMLS) Metathesaurus directly into the model's pre-training process.

Methods

The researchers introduced two primary innovations to the standard BERT architecture to incorporate structured knowledge:

1. Semantic Group Embeddings

UmlsBERT adds a fourth embedding layer: the Semantic Group embedding. Using tools like cTakes, medical terms in the text are identified and mapped to their corresponding UMLS semantic categories. This forces the model to consider the categorical meaning of a word (e.g., "Anatomy" or "Finding") from the very first layer.

Comparison of BERT and UmlsBERT input embedding layers, showing the addition of Semantic Group Embeddings in UmlsBERT.

2. Multi-label CUI Loss

Standard BERT learns by masking a word and trying to guess it. UmlsBERT "softens" this task by identifying words that share the same Concept Unique Identifier (CUI) in UMLS. If the masked word is "lungs," and the UMLS identifies that "lung" and "pulmonary" share the same CUI, the model is trained to predict all of those related terms simultaneously. This effectively teaches the model synonymy during the pre-training phase.

The model was initialized with weights from Bio_ClinicalBERT and then further trained on clinical notes from the MIMIC-III dataset.

Results and Discussion

UmlsBERT outperformed BERT-base, BioBERT, and Bio_ClinicalBERT in 4 out of 5 benchmark tasks. Notably, it achieved 83.0% accuracy on MedNLI (a significant jump over Bio_ClinicalBERT's 81.2%) and an 88.6 F1 score on i2b2 2010 concept extraction.

Knowledge-Driven Clustering

Qualitative analysis confirms that using CUIs in the loss function allows the model to naturally cluster synonyms together in its vector space. It successfully connects terms like "kidney" to "ren" and "feet" to "pedal"—associations that models trained only on raw text often miss. UMAP visualizations demonstrate that UmlsBERT creates much tighter, more distinct clusters of medical terms based on their semantic groups compared to its predecessors.

UMAP visualizations comparing Bio_ClinicalBERT and UmlsBERT, illustrating the improved semantic clustering in UmlsBERT.

Furthermore, semantic group embeddings act as a safety net for rare words. If the model encounters an obscure medical term, it can still rely on the learned representation of that word's category to make an educated guess about its meaning.

Conclusion

The results suggest that intelligently incorporating structured expert knowledge can be more effective than simply throwing more raw text at a model. While computationally intensive to train, UmlsBERT sets a new standard for clinical NLP. Future work will explore exploiting hierarchical "parent-child" relationships in UMLS to further refine the model's medical understanding.

Summary updated

Abstract

Contextual word embedding models, such as BioBERT and Bio_ClinicalBERT, have achieved state-of-the-art results in biomedical natural language processing tasks by focusing their pre-training process on domain-specific corpora. However, such models do not take into consideration expert domain knowledge. In this work, we introduced UmlsBERT, a contextual embedding model that integrates domain knowledge during the pre-training process via a novel knowledge augmentation strategy. More specifically, the augmentation on UmlsBERT with the Unified Medical Language System (UMLS) Metathesaurus was performed in two ways: i) connecting words that have the same underlying `concept' in UMLS, and ii) leveraging semantic group knowledge in UMLS to create clinically meaningful input embeddings. By applying these two strategies, UmlsBERT can encode clinical domain knowledge into word embeddings and outperform existing domain-specific models on common named-entity recognition (NER) and clinical natural language inference clinical NLP tasks.
BibTeX
@inproceedings{michalopoulos2021umlsbert,
  author = {George Michalopoulos and Yuanxin Wang and Hussam Kaka and Helen H. Chen and Alexander Wong},
  title = {{UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the Unified Medical Language System Metathesaurus}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=TaB4ll5GL2},
}
Figure