TL;DR
This paper introduces UMLS-KGI-BERT, a novel framework that integrates the vast biomedical knowledge within the UMLS Metathesaurus into transformer language models like BERT. By treating knowledge graph triples as training data for masked language modeling, they achieve improved performance on biomedical Named Entity Recognition (NER) tasks, particularly in lower-resource languages.
What Problem Does This Paper Address?
The field of biomedical Natural Language Processing (NLP) faces a significant challenge: while transformer models like BERT excel at capturing patterns in text, they often struggle to incorporate the rich, structured domain knowledge crucial for medical applications. Existing approaches to integrate this knowledge, such as knowledge graph embeddings or architectural modifications, can be complex and not universally applicable. This paper aims to bridge this gap by proposing a simpler, data-centric method to infuse biomedical knowledge directly into the pre-training of transformer models, making them more effective for tasks like Named Entity Recognition (NER) without requiring complex architectural changes.
Methods
The core of this work is a "data-centric" approach to knowledge integration. Instead of modifying the BERT architecture or using complex embedding techniques, the authors treat the Unified Medical Language System (UMLS) knowledge graph as a source of training data. They extract triples (head entity, relation, tail entity) from the UMLS and formulate three new pre-training objectives for BERT:
- Entity Prediction: Masking the tail entity in a triple and training the model to predict it.
- Link Prediction: Creating paths of triples and masking relation tokens, training the model to predict the correct relation.
- Triple Classification: Training the model to distinguish between true and false triples, using various negative sampling strategies.
These knowledge-graph-derived tasks are combined with the standard masked language modeling (MLM) objective in a mixed loss function. This allows the transformer to learn from both the unstructured text (like clinical notes) and the structured biomedical knowledge simultaneously. The authors also introduce bertify_umls, a Python library to facilitate this process.
Datasets
- Training Data:
- European Clinical Case Corpus (E3C) v2.0.0: A multilingual corpus of clinical narratives used for masked language modeling.
- UMLS Metathesaurus (2022AB release): Used to generate knowledge graph triples for the new pre-training objectives.
- Evaluation Datasets (NER tasks):
- French: CAS-POS, CAS-SG, QUAERO-MEDLINE
- Spanish: PharmaCoNER, MEDDOCAN
- English: NCBI-Disease, BioRED-NER, JNLPBA04
Key Insights and Ideas
- Knowledge as Data: The paper demonstrates that knowledge graphs can be effectively leveraged by treating their structured information (triples) as additional training data for transformer models, rather than requiring complex architectural changes.
- Simplicity and Effectiveness: The proposed data-centric approach, using standard cross-entropy loss for KG tasks, is shown to be effective, outperforming baselines and requiring less training data compared to state-of-the-art models.
- Benefit for Lower-Resource Languages: The most significant improvements are observed in French and Spanish tasks, suggesting that this method is particularly beneficial for languages with fewer available biomedical text resources.
- Link Prediction's Importance: Ablation studies indicate that the link prediction task, in particular, contributes significantly to the performance gains.
Key Results
The UMLS-KGI-BERT models consistently show improved performance on various biomedical NER tasks across English, French, and Spanish. For instance, in the French CAS-POS task, the KGI-BERT_FR model achieved a macro F1 score of 92.84, outperforming the DrBERT-4GB baseline (90.84). Similarly, in Spanish, the +UMLS-KGI model for BioRoberta-ES achieved 83.90 on PharmaCoNER, surpassing the BioRoberta-ES baseline (80.41). The paper highlights that these gains are achieved with a more data-efficient pre-training strategy.
Limitations and Future Work
Limitations: * The evaluation is primarily limited to token classification NER tasks. A broader range of information extraction and reasoning tasks would provide a more comprehensive understanding of the method's utility. * The current work focuses on mid-to-high-resource languages. Further validation is needed for low-resource settings to fully demonstrate the potential for reducing reliance on large training corpora.
Future Work: * Incorporate a wider variety of graph-based reasoning tasks into the pre-training strategy. * Explore more fine-grained representation of relation types. * Conduct intrinsic evaluations of the learned representations (e.g., embedding visualization). * Investigate the applicability to low-resource languages.
Summary updated