DUVEL: an active-learning annotated biomedical corpus for the recognition of oligogenic combinations

Charlotte Nachtegael, Jacopo De Stefani, Anthony Cnudde, Tom LenaertsDatabase J. Biol. Databases Curation 2024 · DOI

TL;DR

This paper introduces DUVEL, a novel, actively-learned annotated corpus for recognizing "oligogenic combinations" – how multiple gene variants interact to cause disease. By employing active learning, they efficiently created a high-quality dataset that significantly improves the performance of biomedical language models in identifying these complex genetic relationships, paving the way for better disease understanding and curation.

What Problem Does This Paper Address?

Understanding how multiple genetic variants (oligogenic combinations) interact to cause diseases is crucial for advancing our knowledge of disease etiology. While existing biomedical relation extraction (bioRE) datasets have been instrumental in identifying single gene-disease or protein-protein interactions, they are largely insufficient for capturing these more complex, multi-variant relationships. The creation of such datasets is a significant challenge due to the high cost of expert annotation and the sheer volume of unlabelled literature. This paper addresses this gap by developing an efficient method for building a specialized bioRE dataset focused on digenic variant combinations (two genes, two variants) and demonstrating its utility.

Methods

The core of this work lies in the creation of the DUVEL corpus using an active learning (AL) framework. Here's a breakdown of their approach:

  • Data Source: They leveraged the Oligogenic Diseases Database (OLIDA), which curates information on variant combinations associated with genetic diseases, including the source articles.
  • Initial Filtering and Pre-annotation: Articles from OLIDA were filtered to identify those containing digenic variant combinations. Full-text articles were then pre-annotated for genes and variants using the PubTator Central API.
  • Fragment Extraction: Text fragments containing potential digenic variant combinations (gene-variant-gene-variant) were extracted. These fragments were limited to a maximum of 256 tokens to ensure manageable context.
  • Entity Masking: To focus the model on the contextual relationship rather than the entities themselves, gene and variant entities were masked (e.g., @GENE$`, `@VARIANT$).
  • Active Learning for Annotation: Instead of randomly annotating vast amounts of text, they employed an AL strategy. This involved:
  • An initial, partially labelled dataset and a held-out test set were created with randomly selected instances.
  • A biomedical language model (BiomedBERT) was trained on the labelled data.
  • The trained model then identified the most "informative" unlabelled instances (those closest to the decision boundary, where the model is least confident) for human annotation.
  • This cycle of training, selection, and annotation was repeated until a target number of samples (5000) were labelled. This iterative process significantly enriched the dataset with positive instances, addressing class imbalance.
  • Annotation Platform: The ALAMBIC platform was used to facilitate the annotation process.
  • Fine-tuning Experiments: State-of-the-art biomedical language models (BiomedBERT, BiomedBERT-large, BioLinkBERT, and BioM-BERT) were fine-tuned on the DUVEL dataset to evaluate its effectiveness.

Datasets

  • Oligogenic Diseases Database (OLIDA): Used as the primary source for identifying articles relevant to oligogenic variant combinations.
  • DUVEL (Detection of Unique Variant Ensembles in Literature) Corpus: The newly introduced benchmark dataset. It contains 8442 annotated text fragments focusing on digenic variant combinations, with 794 positive instances. This dataset is made publicly available.

Key Insights and Ideas

  • Active Learning is Key for Imbalanced Data: The AL strategy dramatically increased the proportion of positive instances in the training set (from 1.2% to 15%), which is crucial for training models to detect rare but important relationships. This contrasts sharply with random sampling, which yields a highly imbalanced dataset.
  • DUVEL Improves Model Performance: Fine-tuning existing biomedical language models on the DUVEL corpus leads to significant improvements in their ability to recognize digenic variant combinations, demonstrating the dataset's value.
  • Efficiency in Annotation: AL significantly reduces the annotation cost and effort compared to random sampling, making the creation of specialized bioRE datasets more feasible.
  • Focus on Context: Entity masking proved effective in ensuring that models learn to rely on the surrounding text for relation identification, promoting better generalization.

Key Results

  • Dataset Creation: The AL process successfully generated the DUVEL corpus with 8442 fragments, including 794 positive instances, covering 95% of the original annotated articles.
  • Model Performance: After fine-tuning on DUVEL, BiomedBERT-large achieved the highest F1 score of 0.84 for gene-variant pair detection, a substantial improvement over its non-fine-tuned performance.
  • AL vs. Random Sampling: Datasets created with AL showed significantly better performance for models compared to those created with random sampling, highlighting the effectiveness of the AL strategy in producing higher-quality, less imbalanced data.
  • Inter-Annotator Agreement: A substantial agreement (Cohen's kappa of 0.78) was achieved between the primary annotator and a second annotator, indicating the reliability of the annotation guidelines.

Limitations and Future Work

  • Focus on Digenic Combinations: The current DUVEL corpus is limited to digenic variant combinations (two genes, two variants). The authors acknowledge that oligogenic combinations can involve more genes and variants.
  • Fixed Fragment Length: The text fragments were limited to 256 tokens. While this simplifies processing, longer contexts might be beneficial for identifying complex relations.
  • Future Directions: The authors suggest extending DUVEL to include variable n-ary relations and exploring larger text fragment lengths. They also mention the potential for further improvements by adapting AL strategies to background model training and exploring more advanced NLP techniques for handling complex sentence structures and negations.

Summary updated

BibTeX
@article{nachtegael2024duvel,
  author = {Charlotte Nachtegael and Jacopo De Stefani and Anthony Cnudde and Tom Lenaerts},
  title = {{DUVEL: an active-learning annotated biomedical corpus for the recognition of oligogenic combinations}},
  journal = {Database J. Biol. Databases Curation},
  year = {2024},
  doi = {10.1093/database/baae039},
}
Figure