Abstract
Contextual word embedding models, such as BioBERT and Bio_ClinicalBERT, have achieved state-of-the-art results in biomedical natural language processing tasks by focusing their pre-training process on domain-specific corpora. However, such models do not take into consideration expert domain knowledge. In this work, we introduced UmlsBERT, a contextual embedding model that integrates domain knowledge during the pre-training process via a novel knowledge augmentation strategy. More specifically, the augmentation on UmlsBERT with the Unified Medical Language System (UMLS) Metathesaurus was performed in two ways: i) connecting words that have the same underlying 'concept' in UMLS, and ii) leveraging semantic group knowledge in UMLS to create clinically meaningful input embeddings. By applying these two strategies, UmlsBERT can encode clinical domain knowledge into word embeddings and outperform existing domain-specific models on common named-entity recognition (NER) and clinical natural language inference clinical NLP tasks.
Introduction
While general-purpose language models like BERT are powerful, they often struggle with the specialized vocabulary of medicine. Previous attempts to adapt these models, such as BioBERT and Bio_ClinicalBERT, involved further pre-training on massive collections of unstructured biomedical papers and clinical notes. However, these models miss out on decades of human-curated, structured expert knowledge found in medical ontologies. For example, a standard model might not inherently know that "pulmonary" and "lungs" refer to the same concept. UmlsBERT bridges this gap by injecting the Unified Medical Language System (UMLS) Metathesaurus directly into the model's pre-training process.
Methods
The researchers introduced two primary innovations to the standard BERT architecture to incorporate structured knowledge:
1. Semantic Group Embeddings
UmlsBERT adds a fourth embedding layer: the Semantic Group embedding. Using tools like cTakes, medical terms in the text are identified and mapped to their corresponding UMLS semantic categories. This forces the model to consider the categorical meaning of a word (e.g., "Anatomy" or "Finding") from the very first layer.
2. Multi-label CUI Loss
Standard BERT learns by masking a word and trying to guess it. UmlsBERT "softens" this task by identifying words that share the same Concept Unique Identifier (CUI) in UMLS. If the masked word is "lungs," and the UMLS identifies that "lung" and "pulmonary" share the same CUI, the model is trained to predict all of those related terms simultaneously. This effectively teaches the model synonymy during the pre-training phase.
The model was initialized with weights from Bio_ClinicalBERT and then further trained on clinical notes from the MIMIC-III dataset.
Results and Discussion
UmlsBERT outperformed BERT-base, BioBERT, and Bio_ClinicalBERT in 4 out of 5 benchmark tasks. Notably, it achieved 83.0% accuracy on MedNLI (a significant jump over Bio_ClinicalBERT's 81.2%) and an 88.6 F1 score on i2b2 2010 concept extraction.
Knowledge-Driven Clustering
Qualitative analysis confirms that using CUIs in the loss function allows the model to naturally cluster synonyms together in its vector space. It successfully connects terms like "kidney" to "ren" and "feet" to "pedal"—associations that models trained only on raw text often miss. UMAP visualizations demonstrate that UmlsBERT creates much tighter, more distinct clusters of medical terms based on their semantic groups compared to its predecessors.
Furthermore, semantic group embeddings act as a safety net for rare words. If the model encounters an obscure medical term, it can still rely on the learned representation of that word's category to make an educated guess about its meaning.
Conclusion
The results suggest that intelligently incorporating structured expert knowledge can be more effective than simply throwing more raw text at a model. While computationally intensive to train, UmlsBERT sets a new standard for clinical NLP. Future work will explore exploiting hierarchical "parent-child" relationships in UMLS to further refine the model's medical understanding.
Summary updated