KnowledgeHub: An End-to-End Tool for Assisted Scientific Discovery

Shinnosuke Tanaka, James Barry, Vishnudev Kuruvanthodi, Movina Moses, Maxwell J. Giammona, Nathan Herr, Mohab Elkaref, Geeth de MelIJCAI 2024

Introduction

The rapid expansion of scientific literature presents a significant challenge for researchers attempting to stay abreast of new developments. While various AI tools have been developed to assist in this process, the current landscape is highly fragmented. Researchers often find themselves navigating a "bottleneck of effort," jumping between disparate tools for PDF annotation, Named Entity Recognition (NER), and conversational interfaces. KnowledgeHub addresses this fragmentation by providing an integrated, end-to-end pipeline that transforms raw PDF documents into a structured, queryable knowledge base supported by a Retrieval-Augmented Generation (RAG) assistant.

System Architecture and Methods

KnowledgeHub employs a "full-stack" approach to scientific information extraction (IE), organized into three primary stages: ingestion, annotation, and question answering.

The KnowledgeHub system architecture integrates document ingestion, human-in-the-loop annotation, and a RAG-based question-answering system.
  1. Ingestion: The pipeline begins by using GROBID to convert unstructured PDFs into structured XML, extracting metadata and body text. Linguistic processing, including tokenization and Part-of-Speech tagging, is handled by the Stanza library.
  2. Annotation & Ontology: Users can define or import custom ontologies. The platform integrates BRAT, a web-based annotation tool, allowing domain experts to label entities and relationships directly.
  3. Custom Modeling: KnowledgeHub supports training domain-specific NER and Relation Classification (RC) models using PyTorch and BERT-style encoders, ensuring high precision in specialized fields.
  4. Dual-Storage System: Extracted information is stored in a dual-database setup: Neo4j for structured knowledge graphs (triples) and Chroma for vector-based semantic search.
  5. RAG-based QA: Utilizing the IBM Generative AI SDK, the system performs RAG by retrieving relevant text segments and corresponding knowledge graph subgraphs to generate grounded answers to user queries.

Case Study: The Battery Domain

The tool's efficacy was demonstrated through a case study in battery chemistry, utilizing the BattINFO ontology. A dataset of documents ($d^1, d^2, d^3$) was used to evaluate the human-in-the-loop training process, involving approximately 5,000 annotated spans across 150 entity types.

Key Insights: The Human-in-the-Loop Flywheel

A core innovation of KnowledgeHub is the "Auto-Annotation" flywheel. By annotating an initial document and training a preliminary model, researchers can use the AI to suggest annotations for subsequent papers. The human expert then transitions from manual labeling to a more efficient verification and correction role, significantly accelerating the database construction process.

Furthermore, KnowledgeHub enhances model transparency through subgraph grounding. Unlike standard RAG systems that only provide text citations, KnowledgeHub visualizes the specific entities and relations retrieved from the knowledge graph. This allows users to verify the LLM's reasoning against the structured data.

Performance and Results

The researchers evaluated the auto-annotation performance as the training set expanded. The results indicate that the system scales effectively with additional data.

Evaluation of the auto-annotation model's performance (F1 score) in Out-of-Domain (OOD) and In-Domain (ID) settings.

Key findings include:

  • Out-of-Domain (OOD) Performance: A model trained on a single document ($d^1$) achieved a micro F1 score of 54.8% on an unseen document ($d^2$).
  • In-Domain (ID) Scaling: Performance improved as more documents were added to the training set. Training on two documents and testing on a third resulted in an F1 score of 61.9%.
  • Efficiency Gains: The human-in-the-loop approach was found to reduce the time and cost associated with manual annotation by more than 50%.

Limitations and Future Directions

While KnowledgeHub represents a significant step forward, several areas for improvement remain. Currently, the system primarily processes text, leaving valuable data in tables and figures largely inaccessible. Future work aims to integrate multi-modal extraction capabilities. Additionally, the authors plan to explore deeper integration between the Knowledge Graph structure and the LLM's reasoning process, as well as features to support collaborative annotation through inter-annotator agreement metrics.

Summary updated

BibTeX
@inproceedings{tanaka2024knowledgehub,
  author = {Shinnosuke Tanaka and James Barry and Vishnudev Kuruvanthodi and Movina Moses and Maxwell J. Giammona and Nathan Herr and Mohab Elkaref and Geeth de Mel},
  title = {{KnowledgeHub: An End-to-End Tool for Assisted Scientific Discovery}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure