Cross-Domain Limitations of Neural Models on Biomedical Relation Classification

Ilseyar Alimova, Elena Tutubalina, Sergey I. NikolenkoIEEE Access 2022 · DOI · OpenReview

Introduction

In the biomedical domain, Relation Extraction (RE) is a critical task for identifying meaningful connections between entities, such as drug-adverse effect pairs or treatment-disease associations. While state-of-the-art transformer-based models like BioBERT have achieved remarkable success, their performance is typically evaluated in-domain—where training and testing data share the same distribution. This paper investigates the "performance gap" that occurs when these models are applied to out-of-domain data, such as transitioning from scientific PubMed abstracts to clinical electronic health records (EHR).

Methodology

The authors evaluate three distinct neural architectures to assess their robustness across domains:

  1. BioBERT: A transformer-based model pre-trained on large-scale biomedical corpora.
  2. CNN+A: A convolutional neural network integrated with a standard attention mechanism.
  3. LSTM+CA: The proposed model, a Long Short-Term Memory network utilizing a novel Cross-Attention mechanism.
Architecture of the proposed LSTM with Cross-Attention (LSTM+CA) model.

The LSTM+CA model distinguishes itself by employing separate encoders for entities and their surrounding context. As shown in the architecture diagram, the cross-attention mechanism scores context words based on their specific relevance to the entities in question. This design aims to filter out "noisy" text that does not contribute to the relationship, which is vital when moving between the formal language of abstracts and the often fragmented style of clinical notes.

Experimental Setup

The study utilizes four benchmark datasets categorized into two domains:

  • Clinical Records (EHR): The MADE dataset (medications and adverse events) and the i2b2/VA dataset (problems, tests, and treatments).
  • Scientific Abstracts (PubMed): The BC5CDR dataset (chemical-induced diseases) and the PHAEDRA dataset (pharmacovigilance).

To ensure consistency, the authors implemented a standardized method for generating negative examples across all datasets, framing the problem as a binary classification task.

Results and Discussion

The evaluation revealed a significant trade-off between peak performance and generalizability.

In-Domain vs. Cross-Domain Performance

BioBERT emerged as the "specialist," achieving the highest in-domain F1 score of 86.1% on average. However, its performance plummeted by 34.2% when tested on out-of-domain data. In contrast, the proposed LSTM+CA model acted as a "generalist." While its in-domain performance was slightly lower than BioBERT's, it proved more robust, with a smaller cross-domain performance drop of 27.6%. In out-of-domain scenarios, LSTM+CA outperformed BioBERT by an average of 1.6% F1.

Representation Analysis

Using t-SNE visualizations and Euclidean distance measurements, the authors found that LSTM+CA generates denser and more consistent internal representations of context compared to BioBERT. This suggests that BioBERT’s internal states are more sensitive to the specific vocabulary and linguistic nuances of its training set, whereas LSTM+CA focuses on more transferable relational features. Furthermore, the study found a strong inverse relationship between the linguistic "distance" of two datasets and the resulting model performance.

Conclusion

The study highlights a critical limitation in current biomedical RE models: high in-domain accuracy does not guarantee real-world utility across diverse text sources. While BioBERT leads in specialized tasks, the proposed LSTM+CA model offers superior robustness for cross-domain applications. Future research should prioritize "decoupling" models from specific training distributions and establishing cross-domain evaluation as a standard benchmark for biomedical NLP to prevent the overestimation of model capabilities.

Summary updated

Abstract

Relation extraction (RE) aims to extract relational facts from plain text, which is essential to the biomedical research field with the rapid growth of biomedical literature and generally large volumes of biomedicine-related text coming from various sources. Numerous annotated corpora and state-of-the-art models have been introduced in the past five years. However, there are no general guidelines about evaluating models on these corpora in single- and cross-domain settings with diverse entities and relation types. We aim to fill this gap for the task of detecting whether a relation holds between two biomedical entities given a text span. In this work, we present a fine-grained evaluation intended to perform a comparative evaluation of four biomedical benchmarks and understand the efficiency of state-of-the-art neural architectures based on Long Short-Term Memory (LSTM) with cross-attention and Bidirectional Encoder Representations from Transformers (BERT) for relation extraction across two main domains, namely scientific abstracts and electronic health records. We present a comparative evaluation of biomedical RE datasets, including the PHAEDRA, i2b2/VA, BC5CDR, and MADE corpora. Our evaluation of BioBERT and LSTM for binary classification shows significant divergence in in-domain and out-of-domain performance, finding an average drop in F1-measure of 34.2% for BioBERT. The cross-attention LSTM model developed in this work exhibits better cross-domain performance, with a drop of only 27.6% in F-measure.
BibTeX
@article{alimova2022cross,
  author = {Ilseyar Alimova and Elena Tutubalina and Sergey I. Nikolenko},
  title = {{Cross-Domain Limitations of Neural Models on Biomedical Relation Classification}},
  journal = {IEEE Access},
  year = {2022},
  doi = {10.1109/access.2021.3135381},
}
Figure