TL;DR
This paper tackles the challenge of automatically extracting the relationship between reported outcomes and their statistical significance levels from clinical trial publications. They introduce a novel annotated dataset and a machine learning approach, achieving state-of-the-art performance with a fine-tuned BioBERT model, which is crucial for tasks like detecting spin and bias in medical research.
What Problem Does This Paper Address?
Clinical trials are designed to assess the effectiveness of interventions by measuring specific outcomes and using statistical hypothesis testing to determine if observed differences are significant. Significance levels, typically reported as p-values, are crucial indicators of a study's findings. However, extracting this information automatically from the vast and often complex language of scientific publications is challenging. This is particularly important for identifying "spin," where research results are presented in a way that exaggerates beneficial effects, often by focusing on significant outcomes while downplaying non-significant ones, especially for primary outcomes. Previous work has focused on extracting outcomes or significance levels separately, but not the crucial relationship between them.
Methods
The authors propose a three-step algorithm:
- Entity Extraction: Identifying reported outcomes and significance levels. For significance levels, they leverage rule-based methods using regular expressions and sequential rules based on POS-tagging, achieving high precision and recall. For outcomes, they explore various approaches, including rule-based methods, bi-LSTM-CNN-CRF models, and fine-tuned BERT-based models.
- Relation Extraction: Once entities are identified, the core task is to classify pairs of outcomes and significance levels as related or not. They compare several traditional machine learning classifiers (e.g., Random Forest, SVM) using a manually crafted feature set.
- Deep Learning for Relation Extraction: They also employ deep learning by fine-tuning BERT-based models (BERT, SciBERT, BioBERT) for the relation extraction task. This involves inputting sentences with target entities masked and training the models to predict the relationship.
Datasets
The paper introduces a newly annotated corpus for outcome-significance level relations. This corpus is based on their previous work on outcome extraction and contains:
- 663 sentences with annotations for outcome-significance level relations.
- 2,552 annotated relations in total.
- 1,372 positive relations (significance level is related to the outcome).
- 1,180 negative relations (significance level is not related to the outcome).
The corpus is publicly available.
Key Insights and Ideas
- The relationship is key: Simply extracting outcomes and significance levels isn't enough; understanding how they connect is vital for deeper analysis, especially for detecting spin.
- BioBERT shines: Fine-tuning domain-specific language models like BioBERT on lower-cased input proved highly effective for relation extraction, outperforming general BERT and even SciBERT in some configurations.
- Feature engineering matters: Even with deep learning, carefully designed features (like proximity, type of significance expression, and presence of other entities) can significantly boost the performance of traditional classifiers.
- Outcome extraction is a bottleneck: While significance level extraction is highly accurate, outcome extraction remains a challenge, with the best model achieving only ~79% F-measure. This highlights a key area for future improvement.
Key Results
- Significance Level Extraction: Achieved a high F-measure of 97.86% using a rule-based approach.
- Outcome Extraction: The best performance was from a fine-tuned SciBERT uncased model, reaching an F-measure of 79.42%.
- Relation Extraction (Traditional ML): The Random Forest Classifier with tuned hyperparameters achieved the best performance among traditional methods, with an F-measure of 91.33%.
- Relation Extraction (Deep Learning): The fine-tuned BioBERT model (on lower-cased input) achieved the highest F-measure of 94% for relation extraction.
Limitations and Future Work
- End-to-End System: The current approach assumes entities (outcomes and significance levels) are already extracted. Future work includes building an end-to-end system that extracts both entities and relations simultaneously.
- Improving Outcome Extraction: The authors acknowledge that improving the accuracy of outcome extraction is a primary direction for future research, as it directly impacts the relation extraction performance.
- Primary Outcome Identification: A significant future task is developing algorithms to specifically identify the primary outcome and its corresponding significance level, which is crucial for spin detection.
- Confidence Intervals (CIs): The current work focuses on p-values and does not address confidence intervals, which are less frequently reported but still relevant for significance assessment.
Summary updated