TL;DR
This paper tackles the challenge of understanding complex clinical trial reports (CTRs) by developing a system for multi-evidence natural language inference. They leverage the powerful GatorTron language model, fine-tuning it on clinical data and employing a two-stage approach for improved accuracy in determining entailment and retrieving supporting evidence.
What Problem Does This Paper Address?
Keeping up with the ever-growing volume of medical literature is a significant hurdle for healthcare professionals. Computerized Clinical Decision Support Systems (CDSS) aim to help by providing evidence-based recommendations, but their effectiveness hinges on accurately interpreting and retrieving information from complex documents like Clinical Trial Reports (CTRs). SemEval-2023 Task 7 specifically addresses this by focusing on Natural Language Inference (NLI) for clinical trial data, requiring systems to determine if a given statement is entailed or contradicted by information within one or more CTRs, and to identify the specific evidence supporting this decision. This task is particularly challenging due to the need to model long-range dependencies, compare information across multiple documents, and handle numerical reasoning.
Methods
The core of the system is GatorTron, a large, transformer-based language model pre-trained on a massive corpus of biomedical text, including clinical notes and PubMed articles. To adapt GatorTron to the specific domain of clinical trial reports, the authors perform domain adaptation on a curated dataset of ~9,000 CTRs.
The system tackles the task in two sub-tasks:
- Sub-task 1: Textual Entailment: This is approached as a two-stage pipeline. First, an extraction model (GatorTron fine-tuned on sub-task 2 data) identifies relevant sections within a CTR that are likely to support or contradict a given statement. These extracted, relevant sections are then fed into a classification model (a similar GatorTron architecture fine-tuned on sub-task 1 data with unfiltered premises) to determine the final entailment relationship (entailed or contradicted).
- Sub-task 2: Evidence Retrieval: This sub-task is framed as a binary classification problem. For each statement and its corresponding CTR section, the system classifies individual lines of text as either "relevant" or "irrelevant" for determining entailment.
For statements requiring information from two CTRs, the system prepends "first:" or "second:" to the text to preserve the order of the CTRs.
Datasets
The primary dataset used is from SemEval-2023 Task 7, consisting of 1,000 English-language breast cancer CTRs. Each CTR is pre-processed into four sections: eligibility criteria, intervention, results, and adverse events, further subdivided into individual lines. Each data sample includes a statement, the type of support needed (single or multiple CTRs), the relevant CTR section, and a label for entailment (sub-task 1) or relevant line indices (sub-task 2). The dataset is split into training (1,700 statements), validation (200 statements), and test sets (500 statements).
A separate dataset of approximately 9,000 CTRs was used for domain adaptation, filtered to include completed trials with results and focusing on cancer-related studies.
Key Insights and Ideas
- Domain Adaptation is Crucial: While GatorTron is powerful, domain adaptation on clinical trial reports significantly improves performance, suggesting a notable distribution shift between its pre-training data and the target task.
- Evidence Extraction Improves Entailment: A two-stage approach where relevant evidence is first extracted before classification leads to a substantial improvement in sub-task 1 performance compared to using the full CTR premise. This highlights the importance of filtering out noise.
- Annotation Artifacts Matter: The authors found that their system performed surprisingly well on statements alone, suggesting that annotation artifacts in the dataset might be contributing to performance. This is a common issue in NLI datasets and warrants further investigation.
- Numerical Reasoning is a Bottleneck: The system struggles with numerical reasoning, a critical component of many clinical trial reports, indicating an area for future improvement.
Key Results
The system achieved strong results on the SemEval-2023 Task 7:
- Sub-task 1 (Entailment): The final system achieved a 7th place ranking with an F1-score of 0.705. This was a significant improvement over the TF-IDF baseline (0.502) and even outperformed using the full premise (0.637) by using the relevant-only extraction approach (0.696).
- Sub-task 2 (Evidence Retrieval): The system secured 6th place with an F1-score of 0.806.
Limitations and Future Work
- Annotation Artifacts: The authors acknowledge that annotation artifacts may be confounding their results and suggest that adversarial approaches for data augmentation could be beneficial.
- Numerical Reasoning: The system's difficulty with numerical reasoning is a significant limitation. Future work could focus on enhancing the numerical reasoning capabilities of language models for this task.
- Model Size vs. Performance: While larger models generally perform better, the authors note that model size alone isn't sufficient for top performance, emphasizing the importance of domain adaptation and task-specific fine-tuning.
Summary updated