Automation of Citation Screening for Systematic Literature Reviews using Neural Networks: A Replicability Study

Wojciech Kusa, Allan Hanbury, Petr KnotharXiv 2022 · arXiv

TL;DR

This study attempts to replicate two pioneering deep learning approaches for automating citation screening in systematic reviews. The authors found that while one model was successfully replicated, the other suffered from high variance and poor performance, ultimately demonstrating that a much simpler, faster baseline (fastText) can often outperform complex neural architectures in this domain.

What Problem Does This Paper Address?

Systematic Literature Reviews (SLRs) are the gold standard for evidence-based research, particularly in medicine, but they are notoriously slow. A single review can take over 67 weeks to complete, with the "citation screening" phase—where researchers manually sift through thousands of titles and abstracts to find relevant studies—acting as the primary bottleneck.

While machine learning has been proposed to automate this, deep learning (DL) is a relatively new entrant to the field. This paper tackles the "replicability crisis" in AI by investigating whether the first two major DL papers in this space actually deliver on their promises when tested by an independent team across a wide variety of datasets.

Methods

The authors replicated and compared three distinct approaches:

  1. DAE-FF (Paper A): A complex pipeline using three independent Denoising Autoencoders (DAE) to extract features from Bag-of-Words vectors, which are then fed into a feed-forward (FF) network and a linear SVM classifier.
  2. Multi-Channel CNN (Paper B): An end-to-end deep learning model using pre-trained GloVe word embeddings and parallel convolutional layers to classify text.
  3. fastText (Baseline): A shallow neural network that simply averages word embeddings. The authors introduced this as a simpler, faster alternative to see if the "deep" in deep learning was truly necessary.

The primary evaluation metric used was Work Saved over Sampling at 95% recall (WSS@95%), which estimates the percentage of papers a human reviewer doesn't have to read while still finding 95% of the relevant studies.

Datasets

The study utilized 23 publicly available datasets (summarized in Table 1 below), categorized into three groups:

  • Drug (15 datasets): Smaller collections related to medication efficacy.
  • Clinical (3 datasets): Medium-sized reviews on clinical outcomes.
  • SWIFT (5 datasets): Substantially larger reviews (up to ~48,000 citations) created using broader search strategies.
Table 1. Statistics of 23 publicly available datasets used in the experiments on automated citation screening for Systematic Literature Reviews.

Key Insights and Ideas

  • The Replicability Gap: The authors successfully replicated Paper A (average delta of 3.59%), but Paper B was a different story. With an average delta of 17.63% and a failure to train on the largest datasets, the CNN-based approach proved difficult to reproduce.
  • Complexity vs. Stability: The Multi-Channel CNN showed significant variance across different runs. As seen in the boxplots (Figure 1), the CNN's performance range was much wider than the DAE-FF or fastText models, suggesting it is highly sensitive to random initialization.
  • Simplicity Wins on Speed: The fastText model was 72 times faster to train than the DAE-FF model and 8 times faster than the CNN, while providing comparable or better results on 18 out of 23 datasets.
  • Deep Learning isn't the SOTA (Yet): Surprisingly, none of the neural models consistently beat the classic statistical log-linear models established in earlier benchmarks (e.g., Howard et al., 2016).
  • Feature Importance: Using "all features" (title, abstract, and metadata) consistently yielded the best results, confirming that metadata like author and journal info provides valuable signals for screening.

![Fig.

  1. Boxplots showing the high variance of the Multi-Channel CNN compared to other models.](/figures/art_55b16f118ff4458ba2078d380d798afd)

Key Results

The quantitative breakdown in Table 2 highlights the mixed success of the replication. While Paper A's results were stable, Paper B's replicated scores were often significantly lower than originally reported.

Table 2. WSS@95% results for replicated models compared with original results and benchmark models. WSS@95% scores are averages across ten validation runs for each of the 23 review datasets. Underlined scores indicate the highest score within the three tested models, bold values indicate the highest score overall.
  • Training Time: For the largest dataset (Transgenerational), DAE-FF took 5 hours to train, while fastText finished in just 78 seconds.
  • Performance: DAE-FF achieved the highest average precision (0.167), but the authors argue that the massive speed advantage of fastText makes it more practical for "active learning" scenarios where a model must be retrained frequently as a reviewer provides feedback.

Limitations and Future Work

The authors noted several hurdles to replicability:

  • Missing Environment Data: The original papers did not specify software versions (e.g., specific TensorFlow/Keras versions), which can drastically change model behavior.
  • Data Availability: The original processed datasets were not provided, forcing the authors to write their own data-gathering scripts, which might introduce slight variations.
  • Metric Flaws: The WSS metric assumes all documents take the same amount of time to read, which isn't true (e.g., a long abstract vs. a short title).

Future work should focus on more robust generative models (like BERT) and improving the transparency of experimental setups by providing requirements.txt or Docker containers.

Summary updated

Abstract

In the process of Systematic Literature Review, citation screening is estimated to be one of the most time-consuming steps. Multiple approaches to automate it using various machine learning techniques have been proposed. The first research papers that apply deep neural networks to this problem were published in the last two years. In this work, we conduct a replicability study of the first two deep learning papers for citation screening and evaluate their performance on 23 publicly available datasets. While we succeeded in replicating the results of one of the papers, we were unable to replicate the results of the other. We summarise the challenges involved in the replication, including difficulties in obtaining the datasets to match the experimental setup of the original papers and problems with executing the original source code. Motivated by this experience, we subsequently present a simpler model based on averaging word embeddings that outperforms one of the models on 18 out of 23 datasets and is, on average, 72 times faster than the second replicated approach. Finally, we measure the training time and the invariance of the models when exposed to a variety of input features and random initialisations, demonstrating differences in the robustness of these approaches.
BibTeX
@misc{kusa2022automation,
  author = {Wojciech Kusa and Allan Hanbury and Petr Knoth},
  title = {{Automation of Citation Screening for Systematic Literature Reviews using Neural Networks: A Replicability Study}},
  year = {2022},
  eprint = {2201.07534},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.07534},
}
Figure