TL;DR
This paper presents a sentence-BERT based model for automatically determining if a statement is supported or contradicted by clinical trial data (textual entailment) and for retrieving the specific sentences that provide this evidence. The model achieved competitive results on the SemEval-2023 Task 7, demonstrating the potential of pre-trained language models for complex biomedical text understanding tasks.
What Problem Does This Paper Address?
In the era of big data, biomedical researchers face a deluge of information, particularly from clinical trials. Staying abreast of the latest findings is crucial but time-consuming. This paper tackles the challenge of automatically processing clinical trial documents to extract key information. Specifically, it addresses two sub-tasks:
- Textual Entailment: Determining whether a given statement about a clinical trial is supported (entailment) or contradicted by the trial's text. This is vital for verifying claims and ensuring accuracy.
- Evidence Retrieval: Identifying the precise sentences within a clinical trial that support the entailment decision. This provides the necessary justification and transparency for automated findings.
The task is complicated by the fact that clinical trials are structured documents with distinct sections (e.g., "Intervention," "Eligibility," "Results," "Adverse Events"), and statements can refer to either a single trial or a comparison between two trials.
Methods
The authors employed a sentence-BERT (SBERT) based approach, treating the problem as a text similarity task. SBERT is a modification of BERT designed for efficient and effective sentence similarity computations, making it well-suited for comparing statements against trial text.
Key methodological aspects include:
- ClinicalBERT Embeddings: They leveraged pre-trained embeddings from ClinicalBERT, a BERT model specifically trained on electronic health records from the MIMIC III database. This domain-specific pre-training likely enhances the model's understanding of biomedical language.
- Siamese and Triplet Networks: SBERT utilizes these network architectures to learn sentence embeddings that capture semantic similarity.
- Concatenation for Comparison: To handle comparison statements, they prepended "primary trial" or "secondary trial" to the respective sentences before feeding them into the model. This explicitly signals the context of the sentences.
- Two-Epoch Training: Due to time constraints, they trained their model for only one or two epochs.
- Task-Specific Prediction Derivation:
- Task 1 (Entailment): The final label was determined by averaging confidence scores for "Entailment" and "Contradiction" labels provided by SBERT and selecting the label with the highest score.
- Task 2 (Evidence Retrieval): Sentences were selected as evidence if their associated label's confidence score matched the predicted label for Task 1. Specific rules were applied to handle cases where the highest confidence score was "none" or when comparing confidence scores.
Datasets
The primary dataset used was the NLI4CT dataset released for SemEval-2023 Task 7. This dataset comprises clinical trials, each divided into four sections, and "Premise-Statement-Evidence" items. These items can be categorized as "Single" (referring to one trial) or "Comparison" (referring to two trials).
For training and development, the authors split the provided train.json file into 90% training and 10% development sets. The official dev.json was used for testing.
Additionally, they conducted an additional evaluation on animal studies using text passages from the Animal Study Registry (ASR). They manually created a small dataset of 31 studies for this evaluation.
Key Insights and Ideas
- Domain-Specific Pre-training is Key: Using ClinicalBERT embeddings, which are tailored to biomedical text, provided a strong foundation for the model's performance.
- SBERT's Efficacy for Similarity: The choice of SBERT proved effective for the text similarity nature of both entailment and evidence retrieval tasks.
- Handling Comparisons: The simple yet effective strategy of prepending "primary trial" and "secondary trial" to sentences in comparison statements was crucial for the model to differentiate between the two trials.
- Task 2 is Significantly Harder: The results indicate that evidence retrieval (Task 2) is a more challenging task than simple entailment classification (Task 1), likely due to the need for precise sentence identification.
Key Results
- SemEval-2023 Task 7: The best results achieved on the official test sets were f-scores of 0.64 for Task 1 (Textual Entailment) and 0.67 for Task 2 (Evidence Retrieval).
- Epoch Impact: Training with two epochs generally improved results for Task 2, but slightly worsened them for Task 1 compared to one epoch.
- Animal Studies Evaluation: Preliminary evaluation on animal studies suggested that the trained models could potentially be adapted for similar tasks in that domain, though further improvement is needed.
Limitations and Future Work
- Limited Training Time: The authors acknowledge that due to time constraints, they did not perform a 10-fold cross-validation during training, nor did they conduct extensive hyperparameter tuning. This suggests that further optimization could lead to improved performance.
- Potential for Improvement: The authors note that for the animal studies evaluation, the confidence scores assigned to sentences did not always align with the statements, indicating room for improvement in their system's ability to precisely link evidence to claims.
- Future Directions: The paper implicitly suggests that more extensive training, hyperparameter optimization, and potentially more sophisticated model architectures could further enhance performance on both clinical trial and animal study data. The successful preliminary evaluation on animal studies also opens avenues for applying these models to a broader range of biomedical text analysis tasks.
Summary updated