Scholarly Question Answering using Large Language Models in the NFDI4DataScience Gateway

Hamed Babaei Giglou, Tilahun Abedissa Taffa, Rana Abdullah, Aida Usmanova, Ricardo Usbeck, Jennifer D'Souza, Sören AuerarXiv 2024 · arXiv

Finding the right scientific paper in an era of information overload is like searching for a needle in a haystack—except the haystack is growing by millions of documents every year. While federated search engines help by aggregating results from multiple databases, researchers are still left with the daunting task of filtering through long lists of abstracts.

A new paper by Hamed Babaei Giglou and colleagues introduces a solution: a Retrieval-Augmented Generation (RAG) system built directly into the NFDI4DataScience (NFDI4DS) Gateway. This system doesn't just find papers; it allows you to talk to them.

TL;DR

The authors developed a conversational scholarly Question Answering (QA) system that sits on top of a federated search gateway. By combining an ensemble retrieval model with Large Language Models (LLMs), the system enables researchers to interactively filter and query scientific results across 11 major repositories in real-time.

What Problem Does This Paper Address?

The core challenge is information overload. Scientific data is siloed across various repositories like DBLP, Zenodo, and OpenAlex, each with its own structure and terminology. While "federated search" (searching multiple databases at once) solves the fragmentation problem, it doesn't solve the "relevance" problem. Users still have to manually read through dozens of results to find specific answers.

The authors aim to bridge this gap by creating a scholarly QA system that can extract precise answers from a set of search results, effectively turning a static list of papers into a dynamic, conversational knowledge base.

Methods

The system architecture is split into two main functional blocks: the Gateway and the Scholarly QA.

Fig. 1. A functional view of the NFDI4DS Gateway architecture with scholarly QA application.

The Gateway acts as the data aggregator. It performs:

  • API Orchestration: Querying 11 open-source repositories (e.g., IEEE, Wikidata, ORCID).
  • Mapping & Aggregation: Harmonizing heterogeneous data (e.g., mapping "creator" in one source to "author" in another) using a customized schema.org taxonomy.
  • Entity Resolution: Using the DEDUPE machine learning model to identify and merge duplicate records across different sources.

2. Scholarly QA (RAG)

Once the Gateway retrieves a set of documents, the QA system takes over using a RAG approach:

  • Ensemble Retriever: To find the most relevant snippets, the system uses a weighted ensemble of TF-IDF, SVM, and KNN retrievers, all underpinned by Sentence-BERT embeddings.
  • Generator: The system uses GPT-3.5 (via the LangChain framework). It employs a specific prompt template that forces the model to answer only based on the provided search results to minimize hallucinations. It also maintains a "conversation buffer memory" to handle follow-up questions (e.g., "What is the ORKG?" followed by "How do I use it?").

Datasets

To evaluate the system, the authors constructed two distinct benchmarks:

  1. AI-QA Dataset: 3,298 questions generated by GPT-4-Turbo based on clusters of search results. This tests the system's ability to handle general user-like queries.
  2. Comparison-QA Dataset: 1,354 questions derived from the Open Research Knowledge Graph (ORKG). These questions are highly structured (e.g., "In paper X, what is the research problem?") and serve as a "ground truth" for specific property extraction.

Key Insights and Ideas

  • Ensemble Power: The authors found that a single retrieval method wasn't enough. By combining keyword-based (TF-IDF) and semantic-based (SBERT) methods, the system captures both exact terminology and broader context.
  • The Semantic Ceiling: Interestingly, the analysis showed that while Sentence-BERT is great for general recall, traditional keyword matching (TF-IDF/BM25) remains highly effective for scholarly search where specific technical terms are pivotal.
  • Context is King: The use of a "closed-book" prompt (instructing the LLM not to use outside knowledge) is critical in a scholarly context to ensure that the answers are grounded in the actual retrieved literature rather than the LLM's internal (and potentially outdated) training data.

Key Results

  • Gateway Performance: The federated search successfully retrieves an average of 123 documents in 4.93 seconds. The response time distribution is positively skewed, meaning most queries are handled very quickly.
  • QA Accuracy:
    • On the AI-QA dataset, the system achieved a BLEU-1 score of 38.94% and a BERTScore of 36.81%, showing strong alignment with human-like responses.
  • The Comparison-QA task proved harder, with an Exact Match score of 13.93%. This lower score is attributed to the fact that many answers are buried deep in the full text of papers, whereas the current system primarily indexes abstracts and metadata.

Limitations and Future Work

The authors are transparent about several hurdles:

  • Data Availability: The system struggles when the answer to a question isn't present in the metadata or abstract. Future versions will need to incorporate full-text indexing.
  • Clustering Logic: The current K-means clustering used for question generation sometimes produces "noisy" clusters, which can lead to suboptimal evaluation data.
  • LLM Diversity: The study focused on GPT-3.5. The authors suggest that testing a wider range of open-source LLMs (like Llama or Mistral) specifically fine-tuned for scientific text could improve performance.

By turning federated search into a conversation, this work paves the way for a more intuitive research workflow, where the search engine acts less like a card catalog and more like a knowledgeable research assistant.

Summary updated

Abstract

This paper introduces a scholarly Question Answering (QA) system on top of the NFDI4DataScience Gateway, employing a Retrieval Augmented Generation-based (RAG) approach. The NFDI4DS Gateway, as a foundational framework, offers a unified and intuitive interface for querying various scientific databases using federated search. The RAG-based scholarly QA, powered by a Large Language Model (LLM), facilitates dynamic interaction with search results, enhancing filtering capabilities and fostering a conversational engagement with the Gateway search. The effectiveness of both the Gateway and the scholarly QA system is demonstrated through experimental analysis.
BibTeX
@misc{giglou2024scholarly,
  author = {Hamed Babaei Giglou and Tilahun Abedissa Taffa and Rana Abdullah and Aida Usmanova and Ricardo Usbeck and Jennifer D'Souza and Sören Auer},
  title = {{Scholarly Question Answering using Large Language Models in the NFDI4DataScience Gateway}},
  year = {2024},
  eprint = {2406.07257},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.07257},
}
Figure