Syntax-based Transfer Learning for the Task of Biomedical Relation Extraction

Joël Legrand, Yannick Toussaint, Chedy Raïssi, Adrien CouletEMNLP 2018

TL;DR

This paper explores how to improve biomedical relation extraction (RE) using transfer learning (TL), particularly focusing on how syntactic information can enhance this process. They demonstrate that a syntax-aware model (TreeLSTM) benefits more from TL than a purely convolutional model (MCCNN), especially when dealing with limited biomedical data, and show competitive or state-of-the-art results.

What Problem Does This Paper Address?

Training deep learning models for Natural Language Processing (NLP) tasks, especially in specialized domains like biomedicine, often requires vast amounts of labeled data. Creating these annotated corpora is time-consuming and expensive. Relation Extraction (RE), which aims to identify relationships between entities in text, is particularly affected by this data scarcity in the biomedical field due to the complexity of entities (e.g., genetic variations, diseases) and relationships. Transfer learning (TL) offers a solution by leveraging data from related tasks or domains. However, a key challenge is "negative transfer," where using data from a dissimilar domain can actually hurt performance. This paper investigates how to effectively apply TL for biomedical RE, specifically exploring the role of syntactic features in mitigating negative transfer and improving model performance.

Methods

The authors compare two deep learning models for RE:

  • MultiChannel CNN (MCCNN): This model uses multiple CNN channels to process word embeddings, inspired by image processing. It captures different aspects of words but doesn't explicitly incorporate syntactic structure.
  • TreeLSTM: This model is designed to process dependency trees, allowing it to directly leverage the syntactic structure of sentences. This is a key differentiator from the MCCNN.

Both models take word and entity embeddings as input and output a fixed-size sentence representation. A scoring layer then predicts the relationship type between two entities. The core of their approach involves domain adaptation, a type of TL where models are trained on a source domain (larger, more general corpora) and then applied to a target domain (smaller, specific biomedical corpora). They specifically analyze how syntactic similarity between source and target corpora influences the effectiveness of TL with the TreeLSTM model.

Datasets

The paper uses the following datasets:

  • Target Corpora (small, biomedical RE):
    • SNPPhenA: Annotates relationships between single nucleotide polymorphisms (SNPs) and phenotypes from biomedical abstracts.
  • EU-ADR: Annotates relationships between drugs, diseases, and drug targets from PubMed abstracts. This corpus is further divided into subcorpora for drug-disease and drug-target relationships.
  • Source Corpora (larger, general or biomedical RE):
    • SemEval 2013 DDI: Focuses on drug-drug interactions.
  • ADE-EXT: Annotates drugs and conditions (diseases, symptoms) from adverse drug effect reports.
  • reACE: Annotates entities like organizations, persons, and locations from news text, with relationships between them. This is a non-biomedical corpus.

Key Insights and Ideas

  • Syntax matters for Transfer Learning: The TreeLSTM model, which explicitly uses syntactic information, shows significantly better performance gains from transfer learning compared to the MCCNN, which relies solely on word embeddings. This suggests that syntactic similarity is a crucial factor for successful TL in RE.
  • Syntactic Similarity as a Predictor of Transfer Success: The authors propose a method to measure syntactic similarity between corpora using patterns extracted from shortest dependency paths. They found that higher syntactic similarity between a source and target corpus correlates with better transfer learning performance for the TreeLSTM model.
  • General Domain Corpora Can Still Help: Even a non-biomedical corpus like reACE, which has no lexical overlap with the target biomedical corpora, can still provide beneficial syntactic information for the TreeLSTM model. This highlights the power of syntax-agnostic transfer.
  • Lexical Similarity is Less Predictive: The study found that lexical overlap (dictionary coverage) between source and target corpora did not strongly predict transfer learning success. This further emphasizes the importance of syntactic structure.

Key Results

  • Improved Performance: The TreeLSTM model with TL achieved better or state-of-the-art performance on several biomedical RE tasks compared to existing methods and the MCCNN baseline. For instance, for EU-ADR drug-disease, the TreeLSTM model with SemEval 2013 DDI as a source corpus achieved an F1-score of 82.0, outperforming the baseline.
  • Consistent Gains with TreeLSTM: For the TreeLSTM model, adding additional source corpora consistently improved performance, even with dissimilar source data.
  • MCCNN Suffers from Negative Transfer: The MCCNN model, on the other hand, showed slight performance decreases when using additional source corpora, indicating it was more susceptible to negative transfer.
  • Syntactic Similarity Scores: Table 4 shows cosine similarity scores between target and source corpora for different syntactic pattern distributions. For example, the SNPPhenA corpus had a POS-only similarity of 0.80 with the DDI corpus, and this high similarity correlated with good transfer performance.

Limitations and Future Work

  • Pre-trained Embeddings: The authors note that their pre-trained embeddings were derived from biomedical sources, which might influence the performance of TL with non-biomedical corpora like reACE.
  • Future Work: The authors suggest exploring this transfer strategy with Electronic Health Records (EHRs) from various origins, which represent another rich but challenging domain for RE. They also propose further investigation into the interplay of lexical and semantic paradigms with syntactic features.

Summary updated

BibTeX
@inproceedings{legrand2018syntax,
  author = {Joël Legrand and Yannick Toussaint and Chedy Raïssi and Adrien Coulet},
  title = {{Syntax-based Transfer Learning for the Task of Biomedical Relation Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure