DISTANT-CTO: A Zero Cost, Distantly Supervised Approach to Improve Low-Resource Entity Extraction Using Clinical Trials Literature

Anjani Dhrangadhariya, Henning MüllerACL 2022

TL;DR

This paper introduces DISTANT-CTO, a novel, zero-cost approach to automatically extract PICO (Participant, Intervention, Comparator, Outcome) entities from clinical trial literature. By leveraging distant supervision and a unique Gestalt Pattern Matching technique on the vast ClinicalTrials.gov database, they create a massive, weakly-labeled dataset that significantly improves the performance of named entity recognition (NER) models, even outperforming those trained on expensive, manually annotated data.

What Problem Does This Paper Address?

Systematic reviews (SRs) are crucial for evidence-based medicine, but manually identifying PICO information from tens of thousands of clinical studies is an extremely time-consuming bottleneck, often taking months for experts. While automated information extraction (IE) can accelerate this, the field has been hampered by a lack of large, diverse, and annotated corpora. Existing datasets, like EBM-PICO, are relatively small and expensive to create, relying on costly crowd-sourcing or expert annotators. Furthermore, manual annotations are static and can be prone to human error and disagreement, especially for complex entities like interventions. This paper aims to overcome these limitations by creating a scalable, zero-cost method to generate a massive weakly-labeled dataset for PICO entity extraction.

Methods

The core of DISTANT-CTO lies in its distant supervision approach, which avoids manual annotation. They leverage the structured and unstructured data available in the ClinicalTrials.gov (CTO) database.

  1. Candidate Generation: They extract intervention names from structured fields (e.g., InterventionName, InterventionOtherName) and pair them with sentences from unstructured text fields (e.g., BriefSummary, DetailedDescription) where these interventions might be mentioned.
  2. Gestalt Pattern Matching (GPM): Instead of relying on traditional rule-based or ontology-based methods, they use a modified GPM algorithm (specifically, Python's difflib.SequenceMatcher) to find matching blocks between the structured intervention names and the target sentences. This algorithm is flexible and can identify partial matches.
  3. Weak Labeling Function (LFds): They define a scoring function (ds) based on the length of matching blocks relative to the source term. They use a threshold of ds = 1.0 for exact matches and relax it to ds ≥ 0.9 to capture "fairly-accurate partial matches." This process generates both positive (matched) and negative (unmatched) annotated sentences.
  4. NER Model Architecture: They train an end-to-end NER model using a SciBERT encoder, followed by a BiLSTM layer for feature transformation, a self-attention layer to weigh POS tags, and a CRF layer for final tag prediction.

Datasets

  • ClinicalTrials.gov (CTO): The primary source for generating the weakly-labeled dataset. The authors downloaded over 350,000 records.
  • DISTANT-CTO: The newly created, large, weakly-labeled dataset containing over 977,682 "Intervention" mentions across more than a million sentences, with 11 semantic types.
  • EBM-PICO gold: A benchmark dataset of 5000 PICO entity/span annotated documents, used for combined training and evaluation.
  • Physio set: A domain-specific test set of 153 PICO entity/span annotated documents from physiotherapy RCTs, used to evaluate generalization.

Key Insights and Ideas

  • Zero-Cost Annotation is Feasible: The paper demonstrates that it's possible to generate a massive, useful dataset for a complex NLP task like entity extraction without any manual annotation effort, by cleverly leveraging existing structured data.
  • GPM's Flexibility: Using Gestalt Pattern Matching, particularly with a relaxed threshold (ds ≥ 0.9), is crucial for capturing a wider range of intervention mentions, even if it slightly impacts precision. This flexibility is key to overcoming the limitations of rigid matching.
  • Weak Supervision Can Outperform Strong: The results show that their weakly-supervised models trained on DISTANT-CTO can outperform sophisticated models trained on the manually annotated EBM-PICO dataset, highlighting the power of scale.
  • Domain Generalization: The approach shows promising generalization capabilities, performing well on a separate physiotherapy benchmark.

Key Results

  • Improved Performance: DISTANT-CTO models achieved a 2% F1 improvement on the "Intervention" entity extraction task on the EBM-PICO benchmark when trained solely on their weakly-labeled data.
  • Combined Training Boost: When combined with the EBM-PICO training set, their models achieved a significant 5% F1 improvement on the EBM-PICO benchmark.
  • Massive Dataset: The DISTANT-CTO dataset contains over 977,682 token-level "Intervention" annotations, significantly larger than existing resources.
  • Recall vs. Precision Trade-off: Relaxing the ds threshold from 1.0 to ≥ 0.9 increased recall by 3% but slightly decreased precision on their validation set.

Limitations and Future Work

  • False Negatives: A significant portion of false negatives were due to missed intervention abbreviations, partial mentions, or variations in naming conventions, particularly for non-drug composite interventions.
  • Order-Free Matching: The current approach focuses on direct matches. Future work could explore order-free string matching algorithms to better capture variations.
  • External Ontologies: Incorporating external ontologies could help address missed synonyms and abbreviations.
  • Comparator Terms: The paper notes that "Comparator" terms were often not explicitly structured, suggesting the need for a general comparator term dictionary.
  • Extending to Other PICO Entities: The authors plan to extend their approach to automatically generate annotations for Participant (P), Outcome (O), and Study type (S) entities.
  • Exploring Lower ds Thresholds: The paper suggests exploring the quality of DISTANT-CTO for ds < 0.9.

Summary updated

BibTeX
@inproceedings{dhrangadhariya2022distant,
  author = {Anjani Dhrangadhariya and Henning Müller},
  title = {{DISTANT-CTO: A Zero Cost, Distantly Supervised Approach to Improve Low-Resource Entity Extraction Using Clinical Trials Literature}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure