EPICURE Ensemble Pretrained Models for Extracting Cancer Mutations from Literature

Jiarun Cao, Elke M. van Veen, Niels Peek, Andrew G. Renehan, Sophia Ananiadou

TL;DR

This paper introduces EPICURE, a novel approach for automatically identifying cancer-causing mutations from scientific literature. By combining pre-trained language models with ensemble techniques and a smart data augmentation strategy, EPICURE achieves competitive performance and improved generalization across different datasets, addressing the challenge of limited annotated data in this critical domain.

What Problem Does This Paper Address?

Understanding the genetic basis of cancer is crucial for personalized medicine. This involves identifying specific mutations that drive cancer development. While scientific literature is a rich source of this information, automatically extracting it is challenging. Existing methods often struggle with limited, domain-specific datasets, leading to either low performance with deep learning models or reliance on labor-intensive, rule-based systems that lack generalization. This paper tackles the problem of efficiently and accurately extracting cancer mutation information from text, even with scarce annotated data.

Methods

The core of EPICURE lies in its innovative architecture and training strategy:

  • Ensemble Pre-trained Model: EPICURE leverages pre-trained language models (specifically BERT-based models like BioBERT and SciBERT) as its encoder. This allows the model to benefit from vast amounts of general text knowledge.
  • Dual Recognition Patterns: Two distinct patterns are employed for mutation recognition:
  • CRF Pattern: This treats mutation recognition as a sequence-labeling task, assigning labels (like BIO or BEMO) to each token in a sentence. It uses a Conditional Random Field (CRF) layer for this.
  • Span Pattern: This approach directly predicts the start and end positions of mutation mentions using a pointer network-like mechanism.
  • Model Ensemble: The predictions from the CRF and Span patterns are combined using majority voting. This ensemble approach aims to improve robustness and accuracy by leveraging the strengths of both individual patterns.
  • Data Augmentation Strategy: To combat the scarcity of annotated data, EPICURE employs a novel data augmentation technique. It intelligently merges training samples from multiple datasets, carefully filtering negative instances to avoid introducing noise and ensuring that only relevant examples contribute to the expanded training set.

Datasets

The paper evaluates EPICURE on three established benchmark datasets for mutation extraction:

  • TmVar: This dataset is derived from PubMed abstracts and contains a large number of mutation mentions.
  • EMU: This dataset focuses on mutations related to prostate cancer and uses UMLS concepts for filtering.
  • BRONCO: This dataset includes variants and their associated information (genes, diseases, drugs, cell lines) extracted from full-text articles.

The authors highlight that these datasets, while useful, are often limited in scale, motivating their data augmentation approach.

Key Insights and Ideas

  • Pre-trained Models are Key: The paper strongly advocates for the use of pre-trained language models in biomedical text mining, demonstrating their ability to capture rich contextual information even with limited domain-specific fine-tuning data.
  • Ensembling for Robustness: Combining different recognition strategies (CRF and Span) through majority voting significantly boosts the model's overall performance and reliability, suggesting that diverse approaches can complement each other.
  • Smart Data Augmentation: The proposed method for expanding the training set is crucial. By carefully selecting negative examples based on shared vocabulary across datasets, they effectively increase the training data size without introducing detrimental noise, a common pitfall in multi-dataset training.
  • Generalization Matters: The paper emphasizes that existing methods often perform well on their specific training datasets but struggle when applied to new ones. EPICURE's architecture and training strategy aim to overcome this limitation, showing more stable performance across different datasets.

Key Results

  • EPICURE achieves competitive results compared to state-of-the-art baseline models on the TmVar, EMU, and BRONCO datasets.
  • The model achieved the state-of-the-art performance by 1.3% F1 score on the BRONCO dataset.
  • The ablation study showed that both the data augmentation strategy and the model ensemble significantly improved the F1 score across all tested datasets (e.g., 2.2% and 0.9% improvement from data augmentation on TmVar and BRONCO, respectively, and 2.0% and 1.0% improvement from ensemble on TmVar and BRONCO, respectively).
  • The model demonstrates improved generalization capability, maintaining relatively stable results across different datasets, unlike some baseline models that underperform significantly when transferred.

Limitations and Future Work

  • Label Imbalance: The paper acknowledges that EPICURE, like many models, struggles with predicting rare mutation types (e.g., INDEL) due to their scarce frequency in the training data. This label imbalance remains a bottleneck.
  • Future Work: The authors suggest that addressing the label imbalance problem for rare mutation types is a promising direction for future research.

Summary updated

Abstract

To interpret the genetic profile present in a patient sample, it is necessary to know which mutations have important roles in the development of the corresponding cancer type. Named entity recognition is a core step in the text mining pipeline which facilitates mining valuable cancer information from the scientific literature. However, due to the scarcity of related datasets, previous NER attempts in this domain either suffer from low performance when deep learning based models are deployed, or they apply feature based machine learning models or rule based models to tackle this problem, which requires intensive efforts from domain experts, and limit the model generalization capability. In this paper, we propose EPICURE, an ensemble pre trained model equipped with a conditional random field pattern layer and a span prediction pattern layer to extract cancer mutations from text. We also adopt a data augmentation strategy to expand our training set from multiple datasets. Experimental results on three benchmark datasets show competitive results compared to the baseline models.
BibTeX
@inproceedings{cao2021epicure,
  author = {Jiarun Cao and Elke M. van Veen and Niels Peek and Andrew G. Renehan and Sophia Ananiadou},
  title = {{EPICURE Ensemble Pretrained Models for Extracting Cancer Mutations from Literature}},
  booktitle = {CBMS},
  year = {2021},
  doi = {10.1109/cbms52027.2021.00054},
}
Figure