How Much do Knowledge Graphs Impact Transformer Models for Extracting Biomedical Events?

Laura Zanella, Yannick ToussaintACL 2023

TL;DR

This paper investigates how much knowledge graphs (KGs) can boost transformer models for biomedical event extraction. They found that while pre-training transformer models on biomedical data improves trigger detection, integrating KG embeddings significantly enhances argument identification, especially when focusing on the "role" of the argument.

What Problem Does This Paper Address?

Biomedical event extraction is crucial for understanding complex biological processes. It typically involves identifying event triggers (verbs/nouns indicating an event) and their arguments (entities or other events involved). Existing transformer models, while powerful, often struggle with generalization across different biomedical subdomains and can make errors in trigger and argument identification. This paper aims to improve both aspects by exploring the impact of pre-training strategies and the integration of external biomedical knowledge through knowledge graphs.

Methods

The researchers first evaluated five pre-trained transformer models (BERT, BioBERT, SciBERT, PubMedBERT, and BioMedROBERTa) for biomedical event trigger detection. They fine-tuned these models on seven manually annotated biomedical corpora.

For argument identification, they focused on the best-performing model (SciBERT) and enriched its semantic understanding by integrating knowledge graph (KG) embeddings. A KG was constructed from the biomedical corpora, representing entities and triggers as nodes and their relationships (roles) as edges. These KG relationships were then embedded into a low-dimensional vector space. These KG embeddings were integrated with the transformer model's embeddings in various ways to improve argument identification.

Datasets

The study utilized seven publicly available biomedical datasets:

  • Cancer Genetics (CG) 2013
  • Epigenetics and Post-translational Modifications (EPI) 2011
  • GENIA 2011
  • GENIA 2013
  • Infectious Diseases (ID) 2011
  • Pathway Curation (PC) 2013
  • Multi-Level Event Extraction (MLEE)

These datasets were merged and split into training and testing sets. The Cancer Genetics (CG) dataset was specifically used for comparing the final model against baselines.

Key Insights and Ideas

  • Domain-specific pre-training matters for trigger detection: Models pre-trained on biomedical data (like SciBERT and PubMedBERT) generally outperform models pre-trained only on general domain data when it comes to identifying biomedical event triggers.
  • Bi-LSTM classifiers enhance trigger detection: Using a Bi-LSTM classifier on top of transformer models generally leads to better performance in event trigger detection compared to a simple linear classifier.
  • Knowledge graphs are powerful for argument identification: Integrating KG embeddings into transformer models significantly improves the identification of biomedical arguments and their roles.
  • The "role" is key for KG integration: The most substantial performance gains in argument identification were achieved when KG embeddings representing the role of the argument were incorporated, suggesting that understanding the relationship between a trigger and its argument is critical.
  • Generalization is a challenge: Models trained on specific subdomains can have limitations when applied to others, highlighting the need for broader domain understanding.

Key Results

  • Trigger Detection: SciBERT, when fine-tuned with a Bi-LSTM classifier, achieved the highest F1-score (0.72) for biomedical event trigger detection among the evaluated models.
  • Argument Identification: The proposed approach, which integrates KG embeddings into SciBERT, achieved an F1-score of 0.91 for biomedical argument identification. This represents an improvement of around 18% compared to SciBERT without KG embeddings.
  • Baseline Comparison: The final model outperformed two state-of-the-art baselines (TEES-CNN and DeepEventMine) on the CG dataset for both trigger detection (around 1% improvement) and argument identification (around 15-16% improvement).

Limitations and Future Work

  • Pipeline Approach: The current work uses a pipeline approach where trigger detection and argument identification are performed separately. Errors in the first stage are passed to the second. The authors plan to explore joint optimization of both steps to reduce internally transmitted errors.
  • Generalization: While the study aims to improve generalization, the models are still trained on specific corpora. Future work could focus on further enhancing cross-domain generalization.
  • KG Construction: The KG is constructed from the gold data. Exploring methods for automatically constructing or augmenting KGs could be a future direction.

Summary updated

BibTeX
@inproceedings{zanella2023how,
  author = {Laura Zanella and Yannick Toussaint},
  title = {{How Much do Knowledge Graphs Impact Transformer Models for Extracting Biomedical Events?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure