TL;DR
CRUISE-Screening is an open-source, web-based toolbox designed to automate and maintain "living" literature reviews—reviews that stay current as new research is published. By integrating multi-source search APIs with machine learning-assisted screening (including zero-shot LLMs), it streamlines the transition from initial search to final paper selection in a single, unified workflow.
What Problem Does This Paper Address?
For researchers, keeping up with the deluge of new publications is a Herculean task. Traditionally, literature reviews fall into two buckets: systematic reviews, which follow rigid, reproducible protocols (common in medicine), and exploratory reviews, which are more informal and often conducted by students to map out a new field.
Both types suffer from significant friction. Systematic reviews are incredibly tedious and repetitive, while exploratory reviews often lack structured tools, leading to "reinventing the wheel" when different groups repeat the same searches. Furthermore, most existing tools are "static"—once a review is done, it begins to age immediately. The authors address the need for "living" reviews that can be periodically updated and for a tool that bridges the gap between searching for papers and screening them for relevance.
Methods
The authors developed CRUISE-Screening, a web-based application built on Django. Its architecture (seen in Figure 1) is designed to handle the entire lifecycle of a review.
Key components include:
- Multi-Source Search: Instead of relying on a single database, the tool acts as a meta-search engine, querying the APIs of Semantic Scholar, CORE, and PubMed simultaneously. It also includes an internal Elasticsearch-based database for private or specialized collections.
- Structured Protocols: Users define a review protocol (Figure 2) including specific search queries and explicit inclusion/exclusion criteria.
- Hybrid Screening Workflows: The tool offers a "Strict" mode (answering every eligibility question, ideal for systematic reviews) and a "Relaxed" mode (focusing on the final include/exclude decision).
- Automation Methods: To speed up screening, the tool integrates two types of AI:
- Supervised Classification: Uses Logistic Regression (with tf-idf) or fastText. These require a small "seed" of annotated papers (at least 3 included and 3 excluded) to begin making predictions.
- Zero-Shot Question Answering: Leverages Large Language Models (LLMs) like T0 via the HuggingFace library. This approach doesn't require pre-labeled data; instead, it treats each inclusion criterion as a question to be answered based on the paper's abstract.
Datasets
For the demo version of the tool, the authors indexed the DBLP-Citation-network Version 13 collection, which contains a vast repository of computer science citations. For live use, the "dataset" is effectively the real-time results returned by the Semantic Scholar, CORE, and PubMed APIs.
Key Insights and Ideas
- Closing the Loop: Most tools focus either on search (like Google Scholar) or screening (like Rayyan). CRUISE-Screening puts them in one box, allowing a researcher to define a protocol and immediately begin screening the results.
- Living Reviews: The system supports periodic re-running of searches. It intelligently deduplicates results and filters by publication date to ensure the reviewer only sees new, relevant work since their last check.
- LLMs for Screening: The use of zero-shot QA models is a significant shift. By asking an LLM, "Does this paper include a model of knowledge gap?" (as shown in Figure 3), researchers can potentially automate the most time-consuming part of a review without needing to manually label hundreds of training examples first.
- Combating Data Leakage: By allowing researchers to run model predictions before manual screening and then comparing the results later, the tool provides a clean way to evaluate AI performance on truly "unseen" data, avoiding the biases often found in retrospective benchmarks.
Key Results
The primary result is the deployment of the CRUISE-Screening tool itself as a functional, open-source resource.
- Integration: The tool successfully demonstrates the feasibility of a "meta-search" approach for literature reviews, reconciling different data formats from multiple APIs into a single PostgreSQL database.
- Automation Flexibility: The system proves it can support both traditional ML (logistic regression) and modern transformer-based LLMs through a flexible API-based backend.
- User Control: The "Relaxed" vs. "Strict" workflows allow the tool to be useful for both high-stakes medical systematic reviews and quicker scoping reviews for PhD students.
Limitations and Future Work
- API Dependency: As a meta-search engine, the tool is at the mercy of third-party APIs (Semantic Scholar, etc.), which may change or disappear.
- Search Depth: By default, the tool limits searches to the top 500 records per query to maintain speed, which might miss niche papers in very broad fields.
- LLM Hallucinations: The authors warn that LLM-based screening can produce false positives or negatives. They plan to add features that show users the "predicted performance" or confidence level of these models based on past accuracy.
- Cost and Scaling: Running large LLMs is computationally expensive. Future iterations will need to balance the sophistication of the models with the costs of deployment for researchers with limited budgets.
Summary updated