RegulaTome: a corpus of typed, directed, and signed relations between biomedical entities in the scientific literature

Katerina Nastou, Farrokh Mehryary, Tomoko Ohta, Jouni Luoma, Sampo Pyysalo, Lars Juhl JensenbioRxiv 2024 · DOI

TL;DR

This paper introduces RegulaTome, a novel and extensive corpus for biomedical relation extraction, featuring over 16,000 typed, directed, and signed relations across 40+ types, significantly expanding the scope beyond existing datasets. A transformer-based model trained on RegulaTome achieves a promising 66.6% F1-score, demonstrating its effectiveness and paving the way for more sophisticated biomedical text mining.

What Problem Does This Paper Address?

Understanding the complex interplay of biological entities like proteins, chemicals, and their complexes is crucial for advancing biomedical research and applications. Biomedical Natural Language Processing (BioNLP) aims to automate this understanding by extracting relationships between these entities from scientific literature. However, existing datasets for Relation Extraction (RE) often suffer from limitations: they might focus on only one or two types of relations, or restrict annotations to the sentence level. This leaves a gap in comprehensively capturing the multifaceted interactions, especially those involving proteins, protein complexes, families, and chemicals, across entire documents. RegulaTome aims to fill this void by providing a rich, document-level corpus that covers a much broader spectrum of relation types.

Methods

The core of this work is the creation of the RegulaTome corpus. The authors employed a multi-step document selection process, drawing from existing corpora like ComplexTome and PTM triage sets, and also incorporating new data from Reactome. Crucially, they re-annotated these documents to include a wider range of relation types.

For the relation extraction system itself, the authors adapted a previously developed transformer-based architecture. This system is designed for multi-label classification, meaning it can predict multiple relations for a single entity pair. Key features include:

  • Multi-label Classification: Predicts one, several, or no relations for a given entity pair.
  • Directed and Signed Relations: Captures the directionality (e.g., A regulates B) and sign (e.g., positive or negative regulation) of interactions.
  • Transformer Encoder: Leverages pre-trained language models (like RoBERTa) for robust text understanding.
  • Window-based Processing: Handles long biomedical documents by processing text in windows around candidate entity pairs, allowing for cross-sentence relation extraction.
  • Masking Approach: Uses special tokens to highlight the entities of interest within the input window.

The system was trained and evaluated using a document-based split of the RegulaTome corpus, with hyperparameter tuning performed on a development set.

Datasets

The primary dataset introduced in this paper is RegulaTome.

  • RegulaTome: This corpus contains 16,962 relations annotated across 2,521 documents. It covers over 40 relation types, focusing on proteins, chemicals, protein-containing complexes, and protein families. The relations are typed, directed, and signed.
  • Existing Corpora (used for sourcing documents): ComplexTome, BioNLP ST 2009 datasets, Reactome pathway knowledgebase, BioNLP 2010 workshop datasets.

RegulaTome is designed to be more extensive and cover a broader spectrum of relation types than many existing biomedical RE corpora, which often focus on single-type relations at the sentence level.

Key Insights and Ideas

  • The Power of Breadth: The paper strongly suggests that moving beyond single-type, sentence-level relation extraction is essential for truly understanding complex biological systems. RegulaTome's extensive coverage of 40+ relation types is a significant step in this direction.
  • Document-Level Matters: Capturing relations that span across sentences is critical, as many biological interactions are described in a distributed manner within scientific texts. The RegulaTome corpus and the developed system explicitly address this.
  • Multi-label is Key: Biological systems are rarely governed by single, isolated interactions. The ability to assign multiple relation labels to an entity pair reflects this complexity and is crucial for accurate modeling.
  • Transformer Models Shine: The successful application of transformer-based models to this complex, multi-label task highlights their adaptability and power in the biomedical domain.

Key Results

  • Corpus Size: RegulaTome is the most extensive dataset of its kind to date, with 16,962 relations annotated in over 2,500 documents.
  • Relation Diversity: The corpus covers over 40 relation types, significantly broadening the scope compared to many existing RE datasets.
  • Model Performance: A transformer-based model trained on RegulaTome achieved an F1-score of 66.6% on this complex task.
  • Large-Scale Extraction: The trained model was applied to over 36 million PubMed abstracts and 6 million PMC Open Access articles, extracting over 18 million relations.
  • Inter-Annotator Agreement (IAA): The annotation process achieved a high IAA F1-score of 91% for all relation types, indicating high-quality annotations.
  • Error Analysis: Manual error analysis revealed that "ambiguous keyword" and "convoluted text excerpt" were the main sources of model errors, suggesting areas for future improvement in both the corpus and model.

Limitations and Future Work

  • Ambiguity and Complexity: The manual error analysis highlighted that ambiguous keywords and complex sentence structures remain significant challenges for relation extraction models. This suggests that further refinement of annotation guidelines and potentially more sophisticated model architectures are needed.
  • Rare Relations: Performance varies significantly for relations with lower support in the corpus, with some rare relation types achieving 0% F1-score. This indicates a need for strategies to improve the extraction of less frequent interactions.
  • Annotation Errors: The analysis also identified "annotation errors," suggesting that even high-quality corpora can benefit from ongoing refinement and correction.
  • Future Directions: The authors envision RegulaTome contributing to the development of more sophisticated, efficient, and accurate RE systems for biomedical tasks. The large-scale extraction of 18 million relations also opens avenues for downstream analysis and discovery.

Summary updated

Abstract

Abstract Motivation In the field of biomedical text mining, the ability to extract relations from literature is crucial for advancing both theoretical research and practical applications. There is a notable shortage of corpora designed to enhance the extraction of multiple types of relations, particularly focusing on proteins and protein-containing entities such as complexes and families, as well as chemicals. Results In this work we present RegulaTome, a corpus that overcomes the limitations of several existing biomedical relation extraction (RE) corpora, many of which concentrate on single-type relations at the sentence level. RegulaTome stands out by offering 16,962 relations annotated in over 2,500 documents, making it the most extensive dataset of its kind to date. This corpus is specifically designed to cover a broader spectrum of over 40 relation types beyond those traditionally explored, setting a new benchmark in the complexity and depth of biomedical RE tasks. Our corpus both broadens the scope of detected relations and allows for achieving noteworthy accuracy in RE. A Transformer-based model trained on this corpus has demonstrated a promising F1-score (66.6%) for a task of this complexity, underscoring the effectiveness of our approach in accurately identifying and categorizing a wide array of biological relations. This achievement highlights RegulaTome’s potential to significantly contribute to the development of more sophisticated, efficient, and accurate RE systems to tackle biomedical tasks. Finally, a run of the trained relation extraction system on all PubMed abstracts and PMC Open Access full-text documents resulted in over 18 million relations, extracted from the entire biomedical literature. Availability The corpus and all introduced resources are openly accessible via Zenodo ( https://zenodo.org/doi/10.5281/zenodo.10808330 ) and GitHub ( https://github.com/farmeh/RegulaTome_extraction ).
BibTeX
@misc{nastou2024regulatome,
  author = {Katerina Nastou and Farrokh Mehryary and Tomoko Ohta and Jouni Luoma and Sampo Pyysalo and Lars Juhl Jensen},
  title = {{RegulaTome: a corpus of typed, directed, and signed relations between biomedical entities in the scientific literature}},
  year = {2024},
  doi = {10.1101/2024.04.30.591824},
  howpublished = {bioRxiv},
}

Loading related content...

Figure