End-to-End Relation Extraction of Pharmacokinetic Estimates from the Scientific Literature

Ferran Gonzalez Hernandez, Victoria Smith, Quang Nguyen, Palang Chotsiri, Thanaporn Wattanakul, José Antonio Cordero, Maria Rosa Ballester, Albert Sole, Gill Mundin, Watjana Lilaonitkul, Joseph F. Standing, Frank KloproggeACL 2024

TL;DR

This paper introduces PRED, a novel, manually curated dataset for extracting pharmacokinetic (PK) parameters from scientific literature. They also present an end-to-end relation extraction model, achieving high accuracy, which can significantly accelerate the creation of comprehensive PK databases vital for drug development.

What Problem Does This Paper Address?

Drug development relies heavily on understanding how drugs behave in the body – their absorption, distribution, metabolism, and excretion (ADME). This is quantified by pharmacokinetic (PK) parameters. However, this crucial information is scattered across a vast and ever-growing body of scientific literature, often presented in diverse formats. Manually compiling this data into standardized databases is a monumental, time-consuming task, hindering efficient drug discovery and development. Existing databases are limited in scope, and automated methods for extracting this complex information have been lacking, especially for end-to-end extraction of PK parameters and their associated values.

Methods

The authors tackle this challenge with a two-pronged approach:

  1. PRED Corpus Creation: They developed a new, manually annotated dataset called PRED. This corpus contains 3,600 sentences from scientific articles, annotated by domain experts (pharmacometricians and NLP specialists). The annotations cover various PK-related entities (like parameter names, values, units, and comparative terms) and the relationships between them. This meticulous curation ensures high-quality data for training and evaluating models.
  2. End-to-End Relation Extraction Model: They propose a multi-task learning (MTL) architecture based on BioBERT, a language model pre-trained on biomedical text. This model jointly performs Named Entity Recognition (NER) to identify PK-related entities and Relation Extraction (RE) to determine the relationships between these entities. The architecture uses a sequential BIO labeling scheme for NER and a feed-forward layer for relation classification. They also explored the impact of domain-specific pre-training (BioBERT vs. BERTBASE) and the role of local context between entities in the RE process.

Datasets

  • PRED Corpus: A newly introduced, manually curated dataset consisting of 3,600 annotated sentences from scientific articles. It includes 13,404 entity mentions and 12,411 relations. This dataset is publicly available.
  • PubMed: Used as the initial source for retrieving relevant scientific articles.

Key Insights and Ideas

  • The Power of Domain-Specific Pre-training: Using BioBERT, which is pre-trained on biomedical literature, significantly improved performance over a general-domain BERT model, especially for extracting central PK values. This highlights the importance of tailoring language models to specific scientific domains.
  • Joint NER and RE is Beneficial: Training a single model to perform both entity recognition and relation extraction simultaneously yielded better results than optimizing for each task independently. This suggests that sharing learned representations between the tasks is advantageous.
  • Local Context Matters for Relation Extraction: Including local context (text between identified entities) in the relation classifier improved performance for both NER and RE. This indicates that the immediate surrounding text provides crucial clues for understanding relationships.
  • The Challenge of Deviation and Comparison: While central PK values were extracted with high accuracy, extracting deviation values and comparative terms proved more challenging, likely due to their lower frequency and more complex linguistic expression in the literature.

Key Results

  • High Performance: The proposed end-to-end model achieved a micro-average F1-score of 94% for NER and over 85% F1-score across all PK relation types on the PRED test set.
  • BioBERT Advantage: BioBERT as an encoder led to over 3% gains in F1 scores compared to BERTBASE, with a notable ~6% gain for Centralval relations.
  • Annotator Agreement: The model's performance on Centralval relations closely matched or exceeded inter-annotator agreement, demonstrating its competitive accuracy.

Limitations and Future Work

  • Potential Bias in Sentence Selection: The filtering process used to create the PRED corpus might have excluded sentences where the PK NER model missed mentions. This could lead to the extraction of non-PK measurements if the model is deployed without further filtering.
  • Future Directions: The authors suggest exploring different contextual representations for relation extraction and further pre-training on PK-specific literature to potentially improve performance. They also emphasize the need for filtering approaches when deploying such models in production to ensure the extraction of relevant PK measurements.

Summary updated

BibTeX
@inproceedings{hernandez2024end,
  author = {Ferran Gonzalez Hernandez and Victoria Smith and Quang Nguyen and Palang Chotsiri and Thanaporn Wattanakul and José Antonio Cordero and Maria Rosa Ballester and Albert Sole and Gill Mundin and Watjana Lilaonitkul and Joseph F. Standing and Frank Kloprogge},
  title = {{End-to-End Relation Extraction of Pharmacokinetic Estimates from the Scientific Literature}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure