Risk of bias assessment in preclinical literature using natural language processing

Qianying Wang, Jing Liao, Mirella Lapata, Malcolm Macleod

TL;DR

This paper introduces natural language processing (NLP) models to automate the assessment of risk of bias in preclinical research literature, a crucial step for improving research reproducibility and translation. They developed and evaluated various NLP models, including BERT-based approaches, demonstrating significant improvements over traditional methods and offering a publicly available tool for researchers.

What Problem Does This Paper Address?

Systematic reviews are essential for synthesizing evidence, but assessing the risk of bias in the included literature is a time-consuming and error-prone manual process. This is particularly challenging in preclinical research, which often uses different reporting styles and faces unique risks of bias compared to clinical trials. The authors aim to automate this risk of bias assessment to speed up systematic reviews, provide insights for improving research quality, and facilitate the translation of preclinical findings to clinical applications.

Methods

The researchers explored a range of NLP techniques for text classification. They started with baseline models like Support Vector Machines (SVM), Logistic Regression (LogReg), and Random Forests (RF). They then moved to more advanced neural network architectures, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks with Attention (RNN+Attn), and Hierarchical Attention Networks (HANs). Finally, they investigated two strategies using BERT (Bidirectional Encoder Representations from Transformers):

  1. BERT with document chunk pooling (BERT-DCP): Documents were split into smaller chunks, processed by BERT, and then pooled to create a document-level representation.
  2. BERT with sentence extraction (BERT-SE): The most relevant sentences for each risk of bias item were identified using cosine similarity and then fed into a fine-tuned BERT model.

The core idea is to convert text into numerical representations (embeddings) and train models to predict a binary "yes/no" label for each risk of bias item.

Datasets

The study utilized a dataset of 7840 full-text preclinical publications that had been manually annotated for five specific risk of bias items. This dataset was compiled from several sources, including in-house systematic reviews on psychotic disorders and chemotherapy-induced peripheral neuropathy, as well as studies on animal models of various conditions. Additionally, data from the NPQIP (NPG Quality in Publication) and IICARus (Intervention to Improve Compliance with the ARRIVE guidelines) studies were included. The dataset was split into training (80%), validation (10%), and test (10%) sets.

Key Insights and Ideas

  • NLP is a viable solution: The study demonstrates that NLP models can effectively automate risk of bias assessment in preclinical literature, offering a significant improvement over manual methods.
  • Model performance varies by risk of bias item: No single model excels across all risk of bias categories. For instance, BERT-SE was best for animal welfare regulations, while CNNs performed well overall.
  • BERT's potential for specific tasks: While BERT models didn't universally outperform neural networks, BERT-SE showed promise for specific items like animal welfare regulations, highlighting the importance of how information is extracted and presented to the model.
  • The challenge of animal exclusions: The "animal exclusions" category proved particularly difficult, with models achieving relatively low F1 scores, suggesting this is a complex area for automated detection.
  • Importance of context: The limitations of simpler models like bag-of-words and the advantages of contextualized embeddings from BERT underscore the need for models that understand word meaning within its surrounding text.

Key Results

  • Significant improvement over regular expressions: The best NLP models achieved F1 scores that were 13% to 36% higher than the previous regular expression approach for four of the five risk of bias items.
  • Top performance metrics: On the test set, the best models achieved F1 scores of:
  • Random allocation: 82.0%
    • Blinded assessment of outcome: 81.6%
    • Conflict of interests: 82.6%
    • Compliance with animal welfare regulations: 91.4%
    • Animal exclusions: 46.6%
  • Best performing models by category:
    • Convolutional Neural Networks (CNNs) were identified as the overall best models.
  • RNN with attention performed well for random allocation and blinded assessment of outcome.
  • BERT with sentence extraction (BERT-SE) was superior for animal welfare regulations.
  • Publicly available tool: The developed tool and code are publicly available, enabling wider adoption and further research.

Limitations and Future Work

  • Dataset bias: The training dataset was primarily drawn from neuroscience and preclinical studies published in specific journals (e.g., PLoS One, Nature), which might limit the generalizability of the findings to other fields.
  • PDF to text conversion issues: Converting PDFs to plain text can lose document structure and introduce noise (e.g., from figures and tables), impacting model performance. While tools exist to convert to structured XML, they are not always perfect.
  • No "unclear" category: The current models predict binary "yes/no" for risk of bias reporting. The study acknowledges that a more nuanced "unclear" category, common in manual assessments, is not currently supported due to lack of labeled data.
  • Future directions: The authors plan to improve performance by using datasets from more journals and preclinical experiments, and by exploiting structured PubMed XMLs. They also aim to develop approaches for other risk of bias items like sample size calculation and allocation concealment, and to create a user-friendly platform for systematic reviews.

Summary updated

Abstract

We sought to apply natural language processing to the task of automatic risk of bias assessment in preclinical literature, which could speed the process of systematic review, provide information to guide research improvement activity, and support translation from preclinical to clinical research. We use 7840 full-text publications describing animal experiments with yes/no annotations for five risk of bias items. We implement a series of models including baselines (support vector machine, logistic regression, random forest), neural models (convolutional neural network, recurrent neural network with attention, hierarchical neural network) and models using BERT with two strategies (document chunk pooling and sentence extraction). We tune hyperparameters to obtain the highest F1 scores for each risk of bias item on the validation set and compare evaluation results on the test set to our previous regular expression approach. The F1 scores of best models on test set are 82.0% for random allocation, 81.6% for blinded assessment of outcome, 82.6% for conflict of interests, 91.4% for compliance with animal welfare regulations and 46.6% for reporting animals excluded from analysis. Our models significantly outperform regular expressions for four risk of bias items. For random allocation, blinded assessment of outcome, conflict of interests and animal exclusions, neural models achieve good performance; for animal welfare regulations, BERT model with a sentence extraction strategy works better. Convolutional neural networks are the overall best models. The tool is publicly available which may contribute to the future monitoring of risk of bias reporting for research improvement activities.
BibTeX
@inproceedings{wang2021risk,
  author = {Qianying Wang and Jing Liao and Mirella Lapata and Malcolm Macleod},
  title = {{Risk of bias assessment in preclinical literature using natural language processing}},
  booktitle = {Research},
  year = {2021},
  doi = {10.1002/jrsm.1533},
}
Figure