Comparing Encoder-Only and Encoder-Decoder Transformers for Relation Extraction from Biomedical Texts: An Empirical Study on Ten Benchmark Datasets

Mourad Sarrouti, Carson Tao, Yoann Mamy RandriamihajaACL 2022

TL;DR

This paper investigates the effectiveness of different transformer architectures for biomedical relation extraction, a crucial task for knowledge discovery in scientific literature. The authors empirically compare encoder-only models (like BERT) with encoder-decoder models (like T5) across ten benchmark datasets, finding that T5 and multi-task learning significantly improve performance.

What Problem Does This Paper Address?

The scientific literature is a treasure trove of biomedical knowledge, but its sheer volume makes manual curation impossible. Automatically extracting relationships between entities (like drugs, diseases, and proteins) from this literature is essential for advancing research. While pre-trained language models, particularly encoder-only transformers like BERT, have shown promise, the potential of encoder-decoder transformers, specifically T5, for this task remains less explored. This paper aims to bridge this gap by providing a comprehensive comparison of these architectures and exploring how multi-task learning can further enhance performance.

Methods

The core of this research involves an empirical comparison of two main types of transformer architectures for relation extraction:

  • Encoder-Only Transformers: Models like BERT, BioBERT, and PubMedBERT process input text and generate a single representation, which is then used for classification. In this study, a [CLS] token's representation is fed into a softmax layer for relation classification.
  • Encoder-Decoder Transformers: The T5 model, with its encoder-decoder architecture, is designed to generate target tokens based on an input sequence. For relation extraction, T5 is fine-tuned to generate tokens representing the relation types.

Additionally, the authors explore Multi-Task Fine-Tuning (MTFT). This approach involves training a single model on multiple related tasks simultaneously. In this context, T5 is fine-tuned on ten different biomedical relation extraction datasets, allowing it to learn shared features across various entity and relation types. A task-specific token is prepended to the input sequence to indicate the dataset being processed.

Datasets

The study utilizes ten public biomedical relation extraction datasets, covering four major task categories:

  • Protein-Protein Interactions: AIMed, BioInfer, HPRD50, IEPA, LLL
  • Drug-Drug Interactions: DDI (2013 corpus)
  • Chemical-Protein Relationships: ChemProt, DrugProt
  • Disease-Protein Relationships: GAD, EU-ADR

These datasets were preprocessed into a unified format, and entities were anonymized using specific tags (e.g., @PROTEIN$`, `@DRUG$, @CHEMICAL$`, `@GENE$).

Key Insights and Ideas

  • T5 Outperforms BERT Variants: The encoder-decoder architecture of T5 demonstrates superior performance compared to in-domain BERT-based models (BioBERT, PubMedBERT) on average across the ten datasets. This suggests that the generative capabilities of T5 are well-suited for the structured output required in relation extraction.
  • Multi-Task Learning is Powerful: Fine-tuning T5 with multi-task learning (MTFT) leads to substantial performance improvements. This highlights the benefit of learning across related tasks, enabling the model to capture common patterns and generalize better.
  • Domain-Specific Pre-training vs. Architecture: While models pre-trained on biomedical text (like BioBERT and PubMedBERT) generally perform better than general-domain models, T5, even when pre-trained on a general domain, achieves competitive results. This indicates that architectural choices can be as impactful as domain-specific pre-training.
  • SciFive's Performance Anomaly: Interestingly, T5-SciFive, which was pre-trained on biomedical text (PubMed abstracts), did not perform as well as the general-domain T5. This is an unexpected finding that warrants further investigation.

Key Results

The paper reports micro F1 scores for various models on the ten benchmark datasets. Key quantitative findings include:

  • Average Performance: T5 achieved an average micro F1 score of 89.47, outperforming BioBERT (85.44) and PubMedBERT (88.22).
  • T5's Dominance: T5 achieved the highest F1 scores on 5 out of the 10 biomedical RE datasets.
  • MTFT Gains: T5-MTFT achieved the highest average micro F1 score of 91.42, representing an improvement of 1.95 F-score points over the best single-performing model.
  • Significant Gains on Specific Tasks: On the ChemProt dataset, T5-MTFT showed significant improvements of 6.11 and 6.45 F-score points over T5 and BioBERT, respectively, indicating strong knowledge transfer.

Qualitatively, the error analysis revealed common failure modes, such as misclassifications due to repetitive entity mentions, numerical findings, or the presence of both up-regulation and down-regulation words in chemical-protein relation extraction. For drug-drug interactions, the model sometimes confused "Int" (interaction) with "Effect" types.

Limitations and Future Work

The authors acknowledge that while MTFT generally improves performance, there were slight drops on some datasets (e.g., AIMed, BioInfer, GAD). They suggest that the sample size and difficulty of each task/dataset can influence MTFT's effectiveness, potentially leading to underfitting or overfitting.

Future work could involve:

  • Further ablation studies to precisely understand the impact of different biomedical RE tasks and datasets on downstream performance.
  • Investigating the reasons behind T5-SciFive's lower-than-expected performance.
  • Exploring more sophisticated methods for handling complex sentences with multiple entities and ambiguous linguistic cues.

Summary updated

BibTeX
@inproceedings{sarrouti2022comparing,
  author = {Mourad Sarrouti and Carson Tao and Yoann Mamy Randriamihaja},
  title = {{Comparing Encoder-Only and Encoder-Decoder Transformers for Relation Extraction from Biomedical Texts: An Empirical Study on Ten Benchmark Datasets}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure