TL;DR
This paper introduces natural language processing (NLP) models to automate the assessment of risk of bias in preclinical research literature, a crucial step for improving research reproducibility and translation. They developed and evaluated various NLP models, including BERT-based approaches, demonstrating significant improvements over traditional methods and offering a publicly available tool for researchers.
What Problem Does This Paper Address?
Systematic reviews are essential for synthesizing evidence, but assessing the risk of bias in the included literature is a time-consuming and error-prone manual process. This is particularly challenging in preclinical research, which often uses different reporting styles and faces unique risks of bias compared to clinical trials. The authors aim to automate this risk of bias assessment to speed up systematic reviews, provide insights for improving research quality, and facilitate the translation of preclinical findings to clinical applications.
Methods
The researchers explored a range of NLP techniques for text classification. They started with baseline models like Support Vector Machines (SVM), Logistic Regression (LogReg), and Random Forests (RF). They then moved to more advanced neural network architectures, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks with Attention (RNN+Attn), and Hierarchical Attention Networks (HANs). Finally, they investigated two strategies using BERT (Bidirectional Encoder Representations from Transformers):
- BERT with document chunk pooling (BERT-DCP): Documents were split into smaller chunks, processed by BERT, and then pooled to create a document-level representation.
- BERT with sentence extraction (BERT-SE): The most relevant sentences for each risk of bias item were identified using cosine similarity and then fed into a fine-tuned BERT model.
The core idea is to convert text into numerical representations (embeddings) and train models to predict a binary "yes/no" label for each risk of bias item.
Datasets
The study utilized a dataset of 7840 full-text preclinical publications that had been manually annotated for five specific risk of bias items. This dataset was compiled from several sources, including in-house systematic reviews on psychotic disorders and chemotherapy-induced peripheral neuropathy, as well as studies on animal models of various conditions. Additionally, data from the NPQIP (NPG Quality in Publication) and IICARus (Intervention to Improve Compliance with the ARRIVE guidelines) studies were included. The dataset was split into training (80%), validation (10%), and test (10%) sets.
Key Insights and Ideas
- NLP is a viable solution: The study demonstrates that NLP models can effectively automate risk of bias assessment in preclinical literature, offering a significant improvement over manual methods.
- Model performance varies by risk of bias item: No single model excels across all risk of bias categories. For instance, BERT-SE was best for animal welfare regulations, while CNNs performed well overall.
- BERT's potential for specific tasks: While BERT models didn't universally outperform neural networks, BERT-SE showed promise for specific items like animal welfare regulations, highlighting the importance of how information is extracted and presented to the model.
- The challenge of animal exclusions: The "animal exclusions" category proved particularly difficult, with models achieving relatively low F1 scores, suggesting this is a complex area for automated detection.
- Importance of context: The limitations of simpler models like bag-of-words and the advantages of contextualized embeddings from BERT underscore the need for models that understand word meaning within its surrounding text.
Key Results
- Significant improvement over regular expressions: The best NLP models achieved F1 scores that were 13% to 36% higher than the previous regular expression approach for four of the five risk of bias items.
- Top performance metrics: On the test set, the best models achieved F1 scores of:
- Random allocation: 82.0%
- Blinded assessment of outcome: 81.6%
- Conflict of interests: 82.6%
- Compliance with animal welfare regulations: 91.4%
- Animal exclusions: 46.6%
- Best performing models by category:
- Convolutional Neural Networks (CNNs) were identified as the overall best models.
- RNN with attention performed well for random allocation and blinded assessment of outcome.
- BERT with sentence extraction (BERT-SE) was superior for animal welfare regulations.
- Publicly available tool: The developed tool and code are publicly available, enabling wider adoption and further research.
Limitations and Future Work
- Dataset bias: The training dataset was primarily drawn from neuroscience and preclinical studies published in specific journals (e.g., PLoS One, Nature), which might limit the generalizability of the findings to other fields.
- PDF to text conversion issues: Converting PDFs to plain text can lose document structure and introduce noise (e.g., from figures and tables), impacting model performance. While tools exist to convert to structured XML, they are not always perfect.
- No "unclear" category: The current models predict binary "yes/no" for risk of bias reporting. The study acknowledges that a more nuanced "unclear" category, common in manual assessments, is not currently supported due to lack of labeled data.
- Future directions: The authors plan to improve performance by using datasets from more journals and preclinical experiments, and by exploiting structured PubMed XMLs. They also aim to develop approaches for other risk of bias items like sample size calculation and allocation concealment, and to create a user-friendly platform for systematic reviews.
Summary updated