SAFFRON: tranSfer leArning For Food-disease RelatiOn extractioN

Gjorgjina Cenikj, Tome Eftimov, Barbara Koroušić SeljakNAACL 2021

TL;DR

This paper introduces SAFFRON, a novel transfer learning approach for extracting "cause" and "treat" relationships between food and disease entities from biomedical text. By leveraging pre-trained language models and existing biomedical relation extraction datasets, SAFFRON effectively addresses the scarcity of annotated food-disease data, achieving impressive performance.

What Problem Does This Paper Address?

The biomedical domain is awash in data, with millions of research papers and electronic health records. Extracting meaningful relationships, particularly between food and health outcomes, is crucial for understanding nutrition's impact on human health. However, a significant bottleneck is the lack of large, annotated datasets specifically for food-disease relationships. This makes it challenging to build systems that can automatically structure this knowledge for applications like decision support or knowledge graph construction. Existing relation extraction (RE) methods often struggle with this data scarcity, and while transfer learning (TL) offers a promising solution, its application to food-disease RE, especially by repurposing existing biomedical RE resources, is less explored.

Methods

The core of SAFFRON lies in its innovative use of transfer learning with pre-trained transformer models. The authors employ three powerful models: BERT, RoBERTa, and BioBERT. To adapt these models for the specific task of food-disease relation extraction, they introduce a novel data preprocessing pipeline:

  • Entity Masking: Food and disease entities within sentences are replaced with generic placeholders (e.g., XXX, YYY). This encourages the model to learn the contextual cues that signify a relationship, rather than relying on specific entity names.
  • Context Extraction: To further focus the model, only a window of words around the masked entities (5 words before the first entity and 5 words after the last) is retained. This helps in capturing the relevant linguistic features for relation detection.

They then explore two model architectures for classification:

  • Single Sequence Classifier (SSC): Takes the preprocessed sentence as a single input.
  • Sequence Pair Classifier (SPC): Takes the preprocessed sentence (sequence of interest) and concatenates it with 10 randomly sampled sentences that do contain the target relation (ground truth). This approach aims to teach the model what a positive example looks like by comparing the sequence of interest to known positive examples.

Datasets

The research utilizes a combination of existing and newly created datasets:

  • Source Datasets:
    • CrowdTruth: A medical RE dataset containing annotated "cause" and "treat" relations between various biomedical entities.
  • Adverse Drug Events (ADE): A dataset with annotated "cause" (adverse effect) and "dose" relations between drugs and conditions. The "adverse effect" relations are treated as "cause" relations for this study.
  • Target Dataset:
    • FoodDisease: A new dataset created by the authors, consisting of 608 sentences extracted from PubMed abstracts. These sentences were manually annotated for "cause" and "treat" relations between food and disease entities. This dataset is made publicly available.

Key Insights and Ideas

  • Repurposing Biomedical RE for Food-Disease: The central idea is that the linguistic patterns used to express "cause" and "treat" relations are often similar across different biomedical entity types. By training on existing biomedical RE datasets, models can learn these general patterns and apply them to the specific food-disease domain.
  • Contextual Clues Over Entity Names: Masking entities is crucial. It forces the models to learn the how of relation expression (e.g., "leads to," "causes," "treats") rather than just memorizing specific entity pairs. This generalization is key to overcoming data scarcity.
  • BioBERT's Domain Advantage: The domain-specific BioBERT model consistently outperforms general-purpose models like BERT and RoBERTa, highlighting the importance of domain-specific pre-training for biomedical NLP tasks.
  • SSC Outperforms SPC: Surprisingly, the simpler Single Sequence Classifier (SSC) architecture generally achieved better results than the Sequence Pair Classifier (SPC), especially on the source datasets. This suggests that the SPC's method of using sampled ground truths might introduce noise or complexity that hinders performance in some cases.

Key Results

The paper reports impressive macro-averaged F1 scores:

  • On the target FoodDisease dataset (using transfer learning):
    • The best performing models (SSC-BioBERT) achieved macro F1 scores of 0.847 for cause relations and 0.900 for treat relations.
  • On the source datasets (for comparison):
    • The best models (SSC-BioBERT) achieved macro F1 scores of 0.894 for cause relations (CrowdTruth) and 0.871 for treat relations (CrowdTruth).
  • When evaluated on the target FoodDisease dataset without preprocessing, the models achieved lower scores, with the proposed preprocessing leading to an improvement of at least 0.100 in the averaged macro F1 scores for SSC models identifying cause relations.

Limitations and Future Work

  • SPC Performance Anomaly: The paper notes that the SPC models' performance on the "cause" relation in the FoodDisease dataset was notably lower. This is attributed to the small number of positive samples available for constructing ground truth sequences, leading to models that over-predict the negative class.
  • Dataset Uniqueness: The FoodDisease dataset's uniqueness means that if a sentence expresses multiple relations (e.g., a food treats one condition but causes another), only one relation is annotated per sentence. This could potentially penalize models that don't use the proposed preprocessing, as they might output a single prediction for the entire sentence.
  • Future Work: The authors plan to integrate these models into an information extraction pipeline to structure findings from biomedical literature, aiming to facilitate knowledge graph linking between the biomedicine and food/nutrition domains. They also suggest that the developed models could be generalized to other entity types and relation types.

Summary updated

BibTeX
@inproceedings{cenikj2021saffron,
  author = {Gjorgjina Cenikj and Tome Eftimov and Barbara Koroušić Seljak},
  title = {{SAFFRON: tranSfer leArning For Food-disease RelatiOn extractioN}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure