TL;DR
This study attempts to replicate two pioneering deep learning approaches for automating citation screening in systematic reviews. The authors found that while one model was successfully replicated, the other suffered from high variance and poor performance, ultimately demonstrating that a much simpler, faster baseline (fastText) can often outperform complex neural architectures in this domain.
What Problem Does This Paper Address?
Systematic Literature Reviews (SLRs) are the gold standard for evidence-based research, particularly in medicine, but they are notoriously slow. A single review can take over 67 weeks to complete, with the "citation screening" phase—where researchers manually sift through thousands of titles and abstracts to find relevant studies—acting as the primary bottleneck.
While machine learning has been proposed to automate this, deep learning (DL) is a relatively new entrant to the field. This paper tackles the "replicability crisis" in AI by investigating whether the first two major DL papers in this space actually deliver on their promises when tested by an independent team across a wide variety of datasets.
Methods
The authors replicated and compared three distinct approaches:
- DAE-FF (Paper A): A complex pipeline using three independent Denoising Autoencoders (DAE) to extract features from Bag-of-Words vectors, which are then fed into a feed-forward (FF) network and a linear SVM classifier.
- Multi-Channel CNN (Paper B): An end-to-end deep learning model using pre-trained GloVe word embeddings and parallel convolutional layers to classify text.
- fastText (Baseline): A shallow neural network that simply averages word embeddings. The authors introduced this as a simpler, faster alternative to see if the "deep" in deep learning was truly necessary.
The primary evaluation metric used was Work Saved over Sampling at 95% recall (WSS@95%), which estimates the percentage of papers a human reviewer doesn't have to read while still finding 95% of the relevant studies.
Datasets
The study utilized 23 publicly available datasets (summarized in Table 1 below), categorized into three groups:
- Drug (15 datasets): Smaller collections related to medication efficacy.
- Clinical (3 datasets): Medium-sized reviews on clinical outcomes.
- SWIFT (5 datasets): Substantially larger reviews (up to ~48,000 citations) created using broader search strategies.
Key Insights and Ideas
- The Replicability Gap: The authors successfully replicated Paper A (average delta of 3.59%), but Paper B was a different story. With an average delta of 17.63% and a failure to train on the largest datasets, the CNN-based approach proved difficult to reproduce.
- Complexity vs. Stability: The Multi-Channel CNN showed significant variance across different runs. As seen in the boxplots (Figure 1), the CNN's performance range was much wider than the DAE-FF or fastText models, suggesting it is highly sensitive to random initialization.
- Simplicity Wins on Speed: The fastText model was 72 times faster to train than the DAE-FF model and 8 times faster than the CNN, while providing comparable or better results on 18 out of 23 datasets.
- Deep Learning isn't the SOTA (Yet): Surprisingly, none of the neural models consistently beat the classic statistical log-linear models established in earlier benchmarks (e.g., Howard et al., 2016).
- Feature Importance: Using "all features" (title, abstract, and metadata) consistently yielded the best results, confirming that metadata like author and journal info provides valuable signals for screening.

Key Results
The quantitative breakdown in Table 2 highlights the mixed success of the replication. While Paper A's results were stable, Paper B's replicated scores were often significantly lower than originally reported.
- Training Time: For the largest dataset (Transgenerational), DAE-FF took 5 hours to train, while fastText finished in just 78 seconds.
- Performance: DAE-FF achieved the highest average precision (0.167), but the authors argue that the massive speed advantage of fastText makes it more practical for "active learning" scenarios where a model must be retrained frequently as a reviewer provides feedback.
Limitations and Future Work
The authors noted several hurdles to replicability:
- Missing Environment Data: The original papers did not specify software versions (e.g., specific TensorFlow/Keras versions), which can drastically change model behavior.
- Data Availability: The original processed datasets were not provided, forcing the authors to write their own data-gathering scripts, which might introduce slight variations.
- Metric Flaws: The WSS metric assumes all documents take the same amount of time to read, which isn't true (e.g., a long abstract vs. a short title).
Future work should focus on more robust generative models (like BERT) and improving the transparency of experimental setups by providing requirements.txt or Docker containers.
Summary updated