ITTC at SemEval 2023-Task 7: Document Retrieval and Sentence Similarity for Evidence Retrieval in Clinical Trial Data

Rahmad Mahendra, Damiano Spina, Karin VerspoorACL 2023

TL;DR

This paper tackles the challenge of automatically finding evidence within clinical trial reports (CTRs) to support claims made in statements. The authors propose a system that combines document retrieval (using BM25) and sentence similarity (using Word Mover Distance) to identify relevant sentences. Their work highlights the difficulties in retrieving implicit or negative evidence, suggesting avenues for future research with more advanced NLP models.

What Problem Does This Paper Address?

Clinical Trial Reports (CTRs) are crucial for medical practitioners to stay updated on the latest research and provide personalized patient care. However, the sheer volume of published CTRs makes manual review infeasible. Natural Language Inference (NLI) offers a promising solution by helping to automatically extract relevant information. Specifically, this paper focuses on Subtask 2 of the SemEval 2023 Task 7 (NLI4CT) challenge, which aims to identify specific sentences within a CTR that justify a given statement. This is a critical step in building systems that can efficiently synthesize evidence for clinical decision-making.

Methods

The ITTC team approached the evidence retrieval problem using two primary methods:

  • Document Retrieval: They treated the statement as a query and the CTR as a collection of documents (chunks of text, which could be single sentences, multiple sentences, or table cells). They employed the BM25 algorithm, a standard and effective ranking function for information retrieval, to score and retrieve relevant chunks. To improve performance, they pre-processed queries by removing common English stopwords and also excluded frequent terms specific to clinical trial data (e.g., "patient," "trial").
  • Sentence Similarity: For chunks retrieved by BM25, they further assessed their relevance using sentence similarity. They computed the similarity between the statement and each retrieved chunk using Word Mover Distance (WMD). Chunks exceeding a certain similarity threshold were considered evidence.

Additionally, they implemented a "closure strategy" to address low recall. If a query yielded no documents, they considered all chunks as evidence. They also performed sentence decomposition on complex statements, breaking them down into simpler atomic sentences to facilitate better matching. This involved syntactic parsing and coreference resolution.

Datasets

The study utilized the NLI4CT dataset, which consists of clinical trial reports related to breast cancer. This dataset includes statements paired with corresponding CTRs, along with gold-standard labels and evidence explanations. The CTRs are divided into four sections: "Eligibility Criteria," "Intervention," "Results," and "Adverse Events."

Key Insights and Ideas

  • Evidence Retrieval is More Than Keyword Matching: The paper strongly suggests that simply matching keywords isn't enough. Implicit evidence, where the connection isn't explicit, and negative evidence, where the absence of information needs to be confirmed, pose significant challenges.
  • The "Closure Strategy" is a Double-Edged Sword: While the closure strategy dramatically improved recall (by considering all chunks when no relevant ones were found), it came at the cost of precision. This highlights a common trade-off in information retrieval.
  • Domain-Specific Stopwords Matter: Removing terms common in clinical trial reports (beyond general English stopwords) improved the performance of the BM25 retriever.
  • Section-Specific Challenges: The "Eligibility Criteria" section, often containing nuanced conditions, proved harder to retrieve evidence from compared to other sections.

Key Results

The ITTC team submitted six runs combining BM25 and WMD with various configurations.

  • Precision vs. Recall Trade-off: While their system couldn't outperform the baseline's perfect recall, they achieved significantly higher precision. For instance, one of their runs achieved a precision of 0.763 on the test set, compared to the baseline's 0.558.
  • BM25 with Stopword Removal and Closure: The best performing system combined BM25 with stopword removal and the closure strategy, achieving an F1 score of 0.573 on the development set and 0.619 on the test set.
  • Closure Strategy Impact: The closure strategy significantly boosted recall, increasing it from around 0.15 to 0.64 for WMD on the test set, but it lowered precision.
  • Section-Based Performance: The baseline showed perfect recall for "Results" and "Intervention" sections, but struggled with "Eligibility" (precision of 0.195, recall of 1.000).

Limitations and Future Work

The authors acknowledge that their BM25 and WMD approaches struggle with:

  • Medical Vocabulary: Handling domain-specific terminology.
  • Numerical Reasoning: Interpreting and retrieving information involving numbers.
  • Implicit and Negative Evidence: Accurately identifying evidence that is not explicitly stated or confirming the absence of information.

Future work suggested includes:

  • Deep Architectures: Incorporating advanced language models like ColBERT and Dense Passage Retrieval.
  • Linguistic Knowledge: Injecting knowledge about negation, conjunctions, and reasoning capabilities (numerical understanding, comparisons) to improve retrieval accuracy.
  • Subtask Interdependence: Investigating whether NLI tasks inherently require complete evidence retrieval in the clinical trial context.

Summary updated

BibTeX
@inproceedings{mahendra2023ittc,
  author = {Rahmad Mahendra and Damiano Spina and Karin Verspoor},
  title = {{ITTC at SemEval 2023-Task 7: Document Retrieval and Sentence Similarity for Evidence Retrieval in Clinical Trial Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure