Introduction
The rapid expansion of scientific literature presents a significant challenge for researchers attempting to stay abreast of new developments. While various AI tools have been developed to assist in this process, the current landscape is highly fragmented. Researchers often find themselves navigating a "bottleneck of effort," jumping between disparate tools for PDF annotation, Named Entity Recognition (NER), and conversational interfaces. KnowledgeHub addresses this fragmentation by providing an integrated, end-to-end pipeline that transforms raw PDF documents into a structured, queryable knowledge base supported by a Retrieval-Augmented Generation (RAG) assistant.
System Architecture and Methods
KnowledgeHub employs a "full-stack" approach to scientific information extraction (IE), organized into three primary stages: ingestion, annotation, and question answering.
- Ingestion: The pipeline begins by using GROBID to convert unstructured PDFs into structured XML, extracting metadata and body text. Linguistic processing, including tokenization and Part-of-Speech tagging, is handled by the Stanza library.
- Annotation & Ontology: Users can define or import custom ontologies. The platform integrates BRAT, a web-based annotation tool, allowing domain experts to label entities and relationships directly.
- Custom Modeling: KnowledgeHub supports training domain-specific NER and Relation Classification (RC) models using PyTorch and BERT-style encoders, ensuring high precision in specialized fields.
- Dual-Storage System: Extracted information is stored in a dual-database setup: Neo4j for structured knowledge graphs (triples) and Chroma for vector-based semantic search.
- RAG-based QA: Utilizing the IBM Generative AI SDK, the system performs RAG by retrieving relevant text segments and corresponding knowledge graph subgraphs to generate grounded answers to user queries.
Case Study: The Battery Domain
The tool's efficacy was demonstrated through a case study in battery chemistry, utilizing the BattINFO ontology. A dataset of documents ($d^1, d^2, d^3$) was used to evaluate the human-in-the-loop training process, involving approximately 5,000 annotated spans across 150 entity types.
Key Insights: The Human-in-the-Loop Flywheel
A core innovation of KnowledgeHub is the "Auto-Annotation" flywheel. By annotating an initial document and training a preliminary model, researchers can use the AI to suggest annotations for subsequent papers. The human expert then transitions from manual labeling to a more efficient verification and correction role, significantly accelerating the database construction process.
Furthermore, KnowledgeHub enhances model transparency through subgraph grounding. Unlike standard RAG systems that only provide text citations, KnowledgeHub visualizes the specific entities and relations retrieved from the knowledge graph. This allows users to verify the LLM's reasoning against the structured data.
Performance and Results
The researchers evaluated the auto-annotation performance as the training set expanded. The results indicate that the system scales effectively with additional data.
Key findings include:
- Out-of-Domain (OOD) Performance: A model trained on a single document ($d^1$) achieved a micro F1 score of 54.8% on an unseen document ($d^2$).
- In-Domain (ID) Scaling: Performance improved as more documents were added to the training set. Training on two documents and testing on a third resulted in an F1 score of 61.9%.
- Efficiency Gains: The human-in-the-loop approach was found to reduce the time and cost associated with manual annotation by more than 50%.
Limitations and Future Directions
While KnowledgeHub represents a significant step forward, several areas for improvement remain. Currently, the system primarily processes text, leaving valuable data in tables and figures largely inaccessible. Future work aims to integrate multi-modal extraction capabilities. Additionally, the authors plan to explore deeper integration between the Knowledge Graph structure and the LLM's reasoning process, as well as features to support collaborative annotation through inter-annotator agreement metrics.
Summary updated