Extracting Factual Min/Max Age Information from Clinical Trial Studies

Yufang Hou, Debasis Ganguly, Léa Deleris, Francesca Bonin

TL;DR

This paper tackles the challenge of automatically extracting factual minimum and maximum age ranges for participants in clinical trials from research articles. They developed a novel question-answering (QA) based system that leverages structured data from ClinicalTrials.gov for training, outperforming traditional information extraction methods. This work is crucial for building comprehensive knowledge bases about clinical trials, which are essential for evidence-based medicine.

What Problem Does This Paper Address?

Clinical trials are a cornerstone of medical research, providing evidence to guide healthcare practices. However, critical information about these trials, such as participant demographics, is often buried within published research articles, making it difficult to access and synthesize. Specifically, extracting precise minimum and maximum age ranges for study participants from these articles is a significant challenge. This information is vital for understanding the generalizability of trial findings and for building structured databases of clinical trial data. Existing methods struggle with the nuances of natural language, distinguishing factual age ranges from speculative ones, and extracting this information from the full text of articles rather than just abstracts.

Methods

The authors propose a two-stage pipeline for extracting factual min/max age information.

  1. Min/Max Age Sentence Classifier: This component identifies sentences within research articles that likely contain min/max age information. It's trained on data derived from ClinicalTrials.gov, using sentences that include age-related keywords ("age," "ages," "aged," "year," "years") and annotated age values.
  2. Min/Max Age QA Model: This is the core of their novel approach. Instead of traditional pattern matching, they frame the problem as a question-answering task. For each identified sentence, the system answers questions like "What is the min/max age of the participants?" They utilize a Bi-Directional Attention Flow (BiDAF) network, a state-of-the-art QA model, trained on a large dataset of eligibility criteria from ClinicalTrials.gov. This approach is designed to be more robust to variations in linguistic expression compared to rule-based systems.
  3. Non-factual Age Expression Filter: To ensure the extracted ages are factual and not speculative (e.g., "smokers had to be between the ages of 18 and 60 years" for inclusion criteria vs. actual participant ages), they employ a filter that uses a list of "speculation cues" identified in prior work.

Datasets

  • Training Data: The primary source for training their models is the massive structured clinical study records from ClinicalTrials.gov. This dataset, containing eligibility criteria with annotated min/max age values, is used for both the sentence classifier and the QA model. They specifically mention creating 10,000 min age and 10,000 max age pairs for training the QA model.
  • Testing Dataset: A custom-annotated dataset of 50 published journal articles on smoking cessation was used for evaluation. This dataset contains approximately 432,000 tokens and 18,000 sentences, with 843 sentences containing age-related keywords. The annotation was performed by four domain experts.

Key Insights and Ideas

  • Framing Age Extraction as QA: The most significant insight is the successful application of a question-answering framework to extract specific numerical information like age ranges. This moves beyond rigid pattern matching to a more flexible, context-aware approach.
  • Leveraging Structured Data for Distant Supervision: Using ClinicalTrials.gov records as a source of "distant supervision" for training is a clever way to overcome the scarcity of manually annotated clinical trial articles. This allows them to train on a much larger scale than would be feasible with manual annotation alone.
  • Distinguishing Factual vs. Speculative Ages: The explicit inclusion of a non-factual expression filter is crucial. It highlights the challenge of extracting actual participant ages versus inclusion criteria ages, which can differ significantly.
  • BiDAF's Robustness for Numerical Expressions: The paper points out that BiDAF's character-level embeddings help it handle numerical expressions better than standard tokenizers, which is critical for age ranges.

Key Results

The proposed QA-based system (QABasedMinMaxAgeExtractor) significantly outperformed two baselines: a passage retrieval-based IE system and a CRF-based system.

  • Min/Max Age Extraction: The system achieved an F-score of 71.9% for MinAge and 66.7% for MaxAge. This represents a substantial improvement over the baselines, which achieved F-scores of 43.5% and 44.4% (Passage Retrieval) and 45.9% and 21.1% (CRF), respectively.
  • Component Contributions: Ablation studies showed that the QA model (MinMaxAgeQA) had the most impact on performance, while the non-factual filter had a smaller but still positive effect. The sentence classifier (MinMaxAgeSentFinder) had a more pronounced impact on MaxAge extraction.

Limitations and Future Work

  • Noise from Pre-processing: The authors acknowledge that noise introduced during the PDF pre-processing step (e.g., incorrect paragraph segmentation) can lead to articles with annotated min/max ages being predicted as "Null."
  • Speculative Expressions: While they have a filter for non-factual expressions, some speculative sentences that are not captured by their current filter can still lead to incorrect predictions.
  • Introduction Section: Information in the introduction section is excluded due to the risk of including non-study-related age information. Filtering this section is left as future work.
  • Future Work: The authors plan to extend their framework to extract other types of numerical values from clinical research papers, such as outcome values (e.g., abstinence rates) and follow-up time frames (e.g., weeks or months).

Summary updated

Abstract

Population age information is an essential characteristic of clinical trials. In this paper, we focus on extracting minimum and maximum (min/max) age values for the study samples from clinical research articles. Specifically, we investigate the use of a neural network model for question answering to address this information extraction task. The min/max age QA model is trained on the massive structured clinical study records from ClinicalTrials.gov. For each article, based on multiple min and max age values extracted from the QA model, we predict both actual min/max age values for the study samples and filter out non-factual age expressions. Our system improves the results over (i) a passage retrieval based IE system and (ii) a CRF-based system by a large margin when evaluated on an annotated dataset consisting of 50 research papers on smoking cessation.
BibTeX
@inproceedings{hou2019extracting,
  author = {Yufang Hou and Debasis Ganguly and Léa Deleris and Francesca Bonin},
  title = {{Extracting Factual Min/Max Age Information from Clinical Trial Studies}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure