GEAR-Up: Generative AI and External Knowledge-based Retrieval Upgrading Scholarly Article Searches for Systematic Reviews

Kaushik Roy, Vedant Khandelwal, Harshul Surana, Valerie Vera, Amit Sheth, Heather HeckmanarXiv 2023 · arXiv

TL;DR

GEAR-Up is a modular framework that combines Generative AI (ChatGPT) with structured Knowledge Graphs to automate the labor-intensive early stages of systematic literature reviews. By expanding simple user queries into rich, context-aware search strings, the system helps researchers and librarians discover high-quality scholarly articles with significantly less manual effort.

What Problem Does This Paper Address?

Systematic reviews (SRs) are the gold standard for evidence-based research, but they are notoriously slow and resource-heavy. Traditionally, a researcher works with an expert librarian to translate a research question into a complex web of search terms, manually sifting through thousands of papers to find the "sentinel" articles that define a field.

As the volume of published studies grows exponentially, this manual process has become a bottleneck. While Large Language Models (LLMs) like ChatGPT offer a potential solution, they often struggle with "hallucinations" or lack the specific domain depth required for rigorous scholarly work. The authors of GEAR-Up aim to bridge this gap by grounding generative AI in external, factual knowledge bases to assist—rather than replace—the expert librarian.

Methods

The GEAR-Up pipeline is divided into three distinct, modular steps designed to mimic and accelerate the human review process:

  1. Query Expansion Module: The system takes a natural language query and extracts "seed concepts." It then enriches these concepts by querying external Knowledge Graphs (KGs) and pretrained Masked Language Models. This step identifies related terms, definitions, and hidden relationships (e.g., connecting "Hepatitis A" to "Acetaminophen" or "liver carcinoma") that a non-expert might miss.
  2. Additional Related Query Generation Module: These expanded concepts are fed into ChatGPT with a specific prompt to generate a diverse set of reformulated search queries. This leverages the LLM's linguistic flexibility to cover various ways a research question might be phrased in the literature.
  3. Article Search and Retrieval Module: The system executes these queries against scholarly databases like PubMed. To handle the resulting deluge of data, it employs a FAISS-powered (Facebook AI Similarity Search) retriever. This vector-based search narrows the results down to the most relevant titles, abstracts, and specific passages.
Figure 1: (a) shows all the steps in the systematic review process. First, the research question is formulated in the ”Identify the issue and determine the question” phase. Second, the review protocol is determined, i.e., what are the related articles to include based on query concepts and their relation to other concepts. This protocol is used to obtain a targeted search query. Lastly, the targeted search query is executed, the relevant data are extracted, the quality is assessed, and a systematic review is compiled and disseminated. Our proposed system seeks to automate the first two steps of issue identification and review protocol-based targeted query formulation. (b) (1) We use natural language processing tools to obtain seed concepts from the input query. In the figure example, the seed concept is Hepatitis A. (2) We query KGs (e.g., PubMed) using the seed concepts in the input query to obtain additional context. The figure shows subgraphs of concepts connected to the seed concept Hepatitis A and the relationships that connect them. We also query pretrained masked language models to obtain additional context terms. The figure shows the relationships and concepts obtained for the seed concept of Hepatitis A as causes, diagnoses, affects, associated with, complicates, and Acetaminophen. Steps (1) and (2) form the Query Expansion Module from Section 1. (3) The concepts and relationships from the previous steps are fed into ChatGPT with an appropriate prompt to obtain a set of reformulated queries. For example, the prompt in the figure is ”Formulate five prompt queries with the keywords: causes, diagnoses, affects, associated with, complicates, Acetaminophen”. The figure also shows the reformulated queries that ChatGPT generates, e.g., “What are the causes of Hepatitis A and how is it diagnosed?” Step (3) forms the Additional Related Query Generation Module in Section 1. In Step (4), With this list of queries, we first obtain a list of articles using PubMed searches and implement a FAISS-powered retriever that narrows the search to the most relevant articles, including the article titles, abstracts, and relevant passages. This forms the Article Search and Retrieval Module in Section 1. Finally, the list of retrieved outputs is presented to the librarian for feedback-based refinements to the overall system.

Datasets

The system leverages several existing data sources for its operations:

  • PubMed: Used both as the primary database for article retrieval and as a source for medical knowledge expansion.
  • External Knowledge Graphs & MLMs: Used to provide structured context during the expansion phase.
  • Sentinel Articles: For evaluation, the authors used "ground truth" articles provided by an in-house librarian to see if the system could successfully retrieve the same high-quality papers a human expert would select.

Key Insights and Ideas

  • Hybrid Intelligence: The core "aha moment" is the realization that LLMs are best used as creative "reformulators" rather than primary knowledge sources. By grounding the prompt in Knowledge Graph data first, the authors minimize hallucinations and ensure the generated queries are scientifically relevant.
  • Context-Sensitive Safety: The authors address the ethical implications of AI search. They note that "safety" is context-dependent; for instance, a KG path describing how a drug is abused might be restricted for general users but is vital information for an addiction researcher. The modular nature of GEAR-Up allows for these specific safety "guardrails" to be integrated into the Query Expansion phase.
  • Librarian-in-the-Loop: The system is designed for feedback. As shown in Figure 1, the librarian reviews the retrieved articles and provides feedback that can be used to refine the query generation module in real-time.

Key Results

Qualitative evaluations performed alongside an in-house librarian were promising. The system was able to retrieve high-quality articles that closely matched the librarian's own "sentinel" selections.

However, the evaluation also highlighted the nuance required in this field. In one test case (detailed in Figure 2 of the paper), the librarian noted that while the system captured major concepts, it occasionally missed specific sub-concepts or misinterpreted the relationship between them. This reinforces the authors' stance that the tool is a "co-pilot" for librarians, intended to reduce their "bandwidth limitations" rather than operate entirely autonomously.

Limitations and Future Work

The current implementation of GEAR-Up is a proof-of-concept focused primarily on structured knowledge. The authors acknowledge several areas for growth:

  • Data Diversity: Future versions aim to incorporate unstructured data (like full-text documents) and semi-structured data (like web URLs) to further enrich the query expansion.
  • Explainability: A major goal is "tracing back" the system's outputs to their original knowledge sources. This would allow a researcher to see exactly why a specific article was recommended, providing a clear audit trail for the systematic review.
  • Quantitative Scaling: The current evaluation is qualitative; broader, more rigorous benchmarking against standard SR datasets will be necessary to prove the system's efficiency gains at scale.

Summary updated

Abstract

Systematic reviews (SRs) - the librarian-assisted literature survey of scholarly articles takes time and requires significant human resources. Given the ever-increasing volume of published studies, applying existing computing and informatics technology can decrease this time and resource burden. Due to the revolutionary advances in (1) Generative AI such as ChatGPT, and (2) External knowledge-augmented information extraction efforts such as Retrieval-Augmented Generation, In this work, we explore the use of techniques from (1) and (2) for SR. We demonstrate a system that takes user queries, performs query expansion to obtain enriched context (includes additional terms and definitions by querying language models and knowledge graphs), and uses this context to search for articles on scholarly databases to retrieve articles. We perform qualitative evaluations of our system through comparison against sentinel (ground truth) articles provided by an in-house librarian. The demo can be found at: https://youtu.be/zMdP56GJ9mU.
BibTeX
@misc{roy2023gear,
  author = {Kaushik Roy and Vedant Khandelwal and Harshul Surana and Valerie Vera and Amit Sheth and Heather Heckman},
  title = {{GEAR-Up: Generative AI and External Knowledge-based Retrieval Upgrading Scholarly Article Searches for Systematic Reviews}},
  year = {2023},
  eprint = {2312.09948},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.09948},
}
Figure