GEAR-Up: Generative AI and External Knowledge-Based Retrieval: Upgrading Scholarly Article Searches for Systematic Reviews

Kaushik Roy, Vedant Khandelwal, Valerie Vera, Harshul Surana, Heather Heckman, Amit P. ShethAAAI 2024 · DOI

Abstract

Conducting systematic reviews (SRs) is a cornerstone of evidence-based research, yet the process remains notoriously labor-intensive, particularly during the query formulation and article retrieval phases. This paper presents GEAR-Up, a modular pipeline designed to streamline these workflows by combining the creative capabilities of Generative AI (ChatGPT) with the factual grounding of external Knowledge Graphs (KGs). By automating the expansion of seed concepts and the generation of complex search strings, GEAR-Up assists researchers and librarians in discovering relevant scholarly literature more efficiently and reliably. Qualitative evaluations indicate that the system significantly reduces the cognitive load on experts while maintaining high standards for search relevance and reproducibility.

Introduction

Systematic reviews represent the gold standard for synthesizing scientific evidence. However, the "query development" phase—where researchers must craft exhaustive search strings to capture all relevant literature without excessive noise—is a major bottleneck. This phase typically requires deep domain expertise and extensive collaboration with professional librarians. Despite these efforts, the process is often prone to human error, difficult to reproduce, and extremely time-consuming. GEAR-Up addresses these challenges by bridging the gap between a researcher’s natural language inquiry and a professional-grade search strategy through an AI-driven framework.

The GEAR-Up Framework

The GEAR-Up system is structured as a modular pipeline consisting of three primary stages, designed to move from a simple user query to a refined set of research passages.

The GEAR-Up modular pipeline for systematic review query expansion and retrieval.
  1. Query Expansion Module: The process begins with a natural language query from which "seed concepts" (e.g., "Hepatitis A") are extracted. These concepts are then enriched by querying external Knowledge Graphs and pharma-specific ontologies (such as PubMed). This step identifies critical related terms and relationships—such as "causes," "diagnoses," or "complications"—that might be missed in a manual search.
  2. Additional Related Query Generation Module: The expanded concepts and relationships serve as context for ChatGPT. Acting as a brainstorming partner, the LLM generates a diverse set of reformulated search queries. This ensures that the search strategy is comprehensive and captures various linguistic nuances of the research topic.
  3. Article Search and Retrieval Module: The generated queries are executed against the PubMed database. To manage the resulting volume of data, the system utilizes a FAISS-powered (Facebook AI Similarity Search) retriever. This vector-based approach ranks and narrows the results to the most relevant titles, abstracts, and specific passages, facilitating a more targeted review process.

A critical component of this framework is the "Librarian Review" loop. This human-in-the-loop feature allows expert librarians to refine the AI-generated outputs, ensuring that the search strategy remains safe, relevant, and aligned with professional standards.

Knowledge Grounding and Safety

A significant challenge in applying Large Language Models (LLMs) to medical and scientific domains is the risk of "hallucination"—the generation of plausible but factually incorrect information. GEAR-Up mitigates this risk by grounding the LLM in structured Knowledge Graphs. By providing the model with verified keywords and relationships before query generation, the system ensures that the resulting search strings are rooted in established scientific facts.

Furthermore, the authors emphasize that safety in research is context-dependent. For example, information regarding drug abuse might be flagged as unsafe for a general audience but is essential data for an addiction researcher. The GEAR-Up pipeline allows librarians to adjust these guardrails based on the specific context of the systematic review, ensuring that the tool remains flexible yet secure.

Results and Impact

The system was evaluated qualitatively through collaborations with library science experts at the University of South Carolina. Key findings include:

  • Reduced Librarian Burden: Librarians reported that the system produced high-quality articles comparable to manual searches, significantly decreasing the time required for manual query translation and refinement.
  • Improved Reproducibility: The structured nature of the pipeline makes the search strategy transparent and easily replicable by other researchers, a core requirement for high-quality systematic reviews.
  • Functional Feasibility: The successful demonstration of the pipeline—from initial query to FAISS-ranked passages—proves the viability of using vector-based retrieval systems to handle large-scale scholarly data.

Conclusion and Future Work

GEAR-Up demonstrates the potential of integrating Generative AI with structured knowledge bases to automate the most cognitively demanding steps of the systematic review process. While initial qualitative feedback is promising, future work will focus on large-scale quantitative benchmarking using precision and recall metrics. Additionally, the authors aim to explore further automation to reduce the reliance on manual librarian reviews and to integrate the tool into later stages of the SR process, such as data extraction and risk-of-bias assessment.

Summary updated

BibTeX
@inproceedings{roy2024gear,
  author = {Kaushik Roy and Vedant Khandelwal and Valerie Vera and Harshul Surana and Heather Heckman and Amit P. Sheth},
  title = {{GEAR-Up: Generative AI and External Knowledge-Based Retrieval: Upgrading Scholarly Article Searches for Systematic Reviews}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30577},
}
Figure