CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews

Wojciech Kusa, Óscar E. Mendoza, Matthias Samwald, Petr Knoth, Allan Hanbury

Systematic Literature Reviews (SLRs) are the gold standard of evidence-based research, but they are notoriously slow—often taking over a year to complete. The primary bottleneck is Citation Screening (CS), where researchers must manually sift through thousands of titles, abstracts, and eventually full-text papers to find a handful of relevant studies.

While machine learning (ML) and natural language processing (NLP) offer a way to automate this, the field has been hampered by a "dataset gap." Existing datasets are often tiny, poorly documented, lack standardized training/test splits, or suffer from data leakage. Furthermore, most datasets only provide titles and abstracts, ignoring the critical second stage of screening: the full-text article.

Citation Screening Process

To solve this, researchers have introduced CSMeD (Citation Screening Meta-Dataset), a new, large-scale meta-dataset that unifies 325 SLRs into a standardized format for training and evaluating AI models. By harmonizing fragmented data and introducing the first-ever benchmark for full-text screening (CSMeD-FT), this work provides a robust foundation for automating the most time-consuming step of scientific evidence synthesis.

Building the CSMeD Meta-Dataset

The authors developed CSMeD through a rigorous multi-step process:

  • Consolidation & Harmonization: They gathered nine publicly available collections, providing unified access to 325 SLRs across medicine and computer science. Data was converted to the BigBio schema, ensuring interoperability with popular libraries like Hugging Face.
  • Metadata Extension: They scraped the Cochrane Library to enrich the dataset with "SLR protocols"—the specific research questions, search strategies, and eligibility criteria used by human reviewers.
  • Task Reformulation: Instead of treating screening as a simple binary classification, they framed it as a Question Answering (QA) or Natural Language Inference (NLI) task, where the model compares a document against specific eligibility criteria.
  • Full-Text Extraction: For the new CSMeD-FT dataset, they used APIs (SemanticScholar, CORE) and PDF parsing tools (GROBID) to link reviews to the actual full-text content of the primary studies.

This resulted in two primary resources:

  1. CSMeD: The core meta-dataset containing 325 SLRs and over 730,000 documents.
  2. CSMeD-FT: A novel dataset specifically for full-text screening. It includes 213 SLRs where the documents have an average length of over 4,000 words, creating a challenging "long-context" benchmark for modern LLMs.

Key Insights and Model Performance

By deduplicating and providing canonical splits, CSMeD allows for much fairer comparisons between different AI approaches. Testing on this new benchmark revealed several critical insights:

Criteria are Key Neural models perform significantly better when they are given the actual eligibility criteria of the review rather than just the review title. This suggests that models are beginning to "understand" the rules of inclusion.

Zero-Shot Performance Results

In a zero-shot setting (no specific training on the task), the MPNet model using eligibility criteria achieved a TNR@95% of 0.545. This means the model could automatically discard over 54% of irrelevant papers while still catching 95% of the relevant ones.

The Long-Context Hurdle Full-text screening is a different beast than abstract screening. Standard Transformers struggle with the length of full papers. The authors found that specialized models like Clinical-BigBird (designed for long clinical text) are necessary to handle this complexity, significantly outperforming standard Longformers and even zero-shot GPT-3.5 on the CSMeD-FT test set.

LLM Costs While GPT-4 showed high precision on a small subset of full-text documents, the authors noted the prohibitive cost: screening just 500 full-text documents with GPT-4-32k would cost over $400, making specialized, smaller models more practical for large-scale use.

Limitations and Future Work

While CSMeD is a massive leap forward, challenges remain. Extracting protocols from older Cochrane reviews proved difficult due to changing publication standards over the years. Additionally, the current dataset is heavily weighted toward medicine; future versions aim to include more diverse fields like social sciences or engineering.

Ultimately, the authors suggest that the next step is "prospective" benchmarking—evaluating models on reviews that are currently in progress to ensure there is absolutely no chance the model has seen the results during its training phase.

Summary updated

Abstract

Systematic literature reviews (SLRs) play an essential role in summarising, synthesising and validating scientific evidence. In recent years, there has been a growing interest in using machine learning techniques to automate the identification of relevant studies for SLRs. However, the lack of standardised evaluation datasets makes comparing the performance of such automated literature screening systems difficult. In this paper, we analyse the citation screening evaluation datasets, revealing that many of the available datasets are either too small, suffer from data leakage or have limited applicability to systems treating automated literature screening as a classification task, as opposed to, for example, a retrieval or question-answering task. To address these challenges, we introduce CSMeD, a meta-dataset consolidating nine publicly released collections, providing unified access to 325 SLRs from the fields of medicine and computer science. CSMeD serves as a comprehensive resource for training and evaluating the performance of automated citation screening models. Additionally, we introduce CSMeD-FT, a new dataset designed explicitly for evaluating the full text publication screening task. To demonstrate the utility of CSMeD, we conduct experiments and establish baselines on new datasets.
BibTeX
@inproceedings{kusa2023csmed,
  author = {Wojciech Kusa and Óscar E. Mendoza and Matthias Samwald and Petr Knoth and Allan Hanbury},
  title = {{CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
}
Figure