Extracting Factual Min/Max Age Information from Clinical Trial Studies

Yufang Hou, Debasis Ganguly, Léa A. Deleris, Francesca Bonin

TL;DR

This paper tackles the challenge of automatically extracting precise minimum and maximum age ranges of participants from clinical trial research articles. They achieve this by developing a question-answering (QA) based system, trained on a large dataset derived from ClinicalTrials.gov, which significantly outperforms previous methods. This work is crucial for building comprehensive knowledge bases about clinical trials, essential for evidence-based medicine.

What Problem Does This Paper Address?

Clinical trials are a cornerstone of medical evidence, but crucial details about their participants, like age ranges, are often buried within lengthy research papers. Manually extracting this information is time-consuming and prone to errors. While some studies focus on eligibility criteria, these might not reflect the actual age distribution of participants. This paper aims to bridge this gap by accurately extracting factual minimum and maximum age values for study samples directly from the full text of clinical trial articles, distinguishing them from speculative or irrelevant age mentions.

Methods

The authors propose a pipeline approach with three main components:

  1. Min/Max Age Sentence Classifier: This component identifies sentences likely to contain min/max age information. It's trained on sentences extracted from ClinicalTrials.gov that explicitly mention age or years.
  2. Min/Max Age QA Model: This is the core of their system. They frame the extraction task as a question-answering problem, where a neural network model (specifically, a Bi-Directional Attention Flow - BiDAF network) is trained to answer questions like "What is the min/max age of the participants?" based on the identified sentences. A key innovation here is training this QA model on a large dataset derived from ClinicalTrials.gov's "Eligibility Criteria" section, which provides a form of "distant supervision" for factual age ranges.
  3. Non-factual Age Expression Filter: To ensure the extracted ages are factual and relevant to the study participants, this component uses a list of "speculation cues" (e.g., "if," "at least," "must") to filter out age expressions that are not definitive.

Datasets

  • Training Data: The primary source for training their models is the massive structured clinical study records from ClinicalTrials.gov. This dataset, containing eligibility criteria with annotated min/max ages, is used to create training data for both the sentence classifier and the QA model.
  • Testing Dataset: A custom-annotated dataset of 50 published research papers on smoking cessation was used for evaluation. This dataset contains approximately 432k tokens and 18k sentences, with 843 sentences containing age-related keywords.

Key Insights and Ideas

  • QA as a Powerful Extraction Tool: Framing age extraction as a question-answering task, rather than traditional information extraction, proves highly effective, especially when the QA model is trained on relevant, albeit "noisy," data like eligibility criteria.
  • Distant Supervision from ClinicalTrials.gov: Leveraging the structured data in ClinicalTrials.gov as a source of distant supervision for training a QA model is a clever way to overcome the scarcity of manually annotated clinical trial articles.
  • Distinguishing Factual vs. Speculative Age: The explicit focus on filtering out non-factual or speculative age expressions is critical for the reliability of the extracted information, moving beyond simple pattern matching.
  • Full Article Analysis is Necessary: The authors highlight that age information is often not present in abstracts, necessitating the analysis of the entire article.

Key Results

The proposed QA-based system significantly outperforms two baseline methods: a passage retrieval-based IE system and a CRF-based system.

  • Min/Max Age Extraction: The system achieved an F-score of 71.9% for MinAge and 66.7% for MaxAge. This represents a substantial improvement over the baselines, which achieved F-scores of 43.5% and 44.4% for MinAge and MaxAge respectively.
  • Component Contributions: Ablation studies showed that the Min/Max Age QA Model had the most significant impact on performance, while the Non-factual Age Expression Filter also contributed positively. The Min/Max Age Sentence Finder was more impactful for MaxAge extraction.

Limitations and Future Work

  • Preprocessing Noise: The PDF parsing step can introduce noise, potentially leading to missed annotations or incorrect predictions.
  • Uncaptured Speculative Expressions: While a filter for speculative expressions is used, some nuanced or complex speculative sentences might still lead to incorrect predictions. The authors note an instance where a speculative sentence led to an incorrect max age prediction.
  • Future Work: The authors plan to extend their framework to extract other types of numeric values from clinical research papers, such as outcome values (e.g., abstinence rates) and time frames (e.g., follow-up periods). They also suggest further work on filtering unrelated age information from introduction sections.

Summary updated

Abstract

Population age information is an essential characteristic of clinical trials. In this paper, we focus on extracting minimum and maximum (min/max) age values for the study samples from clinical research articles. Specifically, we investigate the use of a neural network model for question answering to address this information extraction task. The min/max age QA model is trained on the massive structured clinical study records from ClinicalTrials.gov. For each article, based on multiple min and max age values extracted from the QA model, we predict both actual min/max age values for the study samples and filter out non-factual age expressions. Our system improves the results over (i) a passage retrieval based IE system and (ii) a CRF-based system by a large margin when evaluated on an annotated dataset consisting of 50 research papers on smoking cessation.
BibTeX
@inproceedings{hou2019extracting,
  author = {Yufang Hou and Debasis Ganguly and Léa A. Deleris and Francesca Bonin},
  title = {{Extracting Factual Min/Max Age Information from Clinical Trial Studies}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure