Abstract
STRUCTSENSE is an open-source, multi-agent framework designed to extract structured data from complex scientific literature. By integrating Large Language Models (LLMs) with domain-specific ontologies and a human-in-the-loop (HIL) feedback mechanism, the framework addresses common challenges such as domain-specific polysemy, brittle extraction pipelines, and lack of grounding. Benchmarked across specialized neuroscience tasks, STRUCTSENSE demonstrates significant improvements in extraction accuracy and reliability, particularly when leveraging HIL feedback to refine model outputs and align them with canonical knowledge bases.
Introduction
The exponential growth of scientific literature—with millions of papers published annually—has made manual data synthesis nearly impossible for researchers. While Large Language Models (LLMs) offer a potential solution for automated extraction, they frequently struggle with highly specialized domains. The authors identify three primary "pain points" in current automated extraction: domain sensitivity to polysemous terms, brittle multi-stage pipelines where early errors propagate, and a lack of grounding to standardized knowledge bases (ontologies).
As illustrated in the table above, a term like "cortex" carries vastly different meanings in neuroscience versus plant biology. Without proper grounding, LLMs are prone to hallucinations or contextual errors that render extracted data unusable for scientific research.
Methodology: The STRUCTSENSE Framework
To address these challenges, the researchers developed STRUCTSENSE, a modular framework powered by four specialized agents that collaborate to ensure high-fidelity extraction and grounding.
The Agentic Workflow
- Extractor Agent: Responsible for converting raw, unstructured text into a structured JSON format based on a predefined schema.
- Alignment Agent: Maps extracted terms to canonical concepts in an ontology database (e.g., Uberon or ADO) using vector storage. This provides the necessary grounding that standard LLMs lack.
- Judge Agent: Acts as an internal critic, evaluating the quality of the extraction and alignment to assign a confidence score.
- Feedback Agent: Incorporates human corrections into the loop, allowing the system to iteratively update or correct extracted terms based on expert input.
This "agentic" approach allows the system to maintain contextual memory across a document and self-correct, rather than treating every sentence as an isolated event.
Experimental Setup and Datasets
The framework was benchmarked across three distinct neuroscience-focused tasks to test its versatility:
- ReproSchema: Converting semi-structured clinical assessment instruments (PDFs) into machine-readable metadata.
- Resource Extraction (BBQS): Identifying models, datasets, and software tools mentioned in full-text papers.
- NER Term Extraction (BICAN): Extracting complex entities like brain regions and experimental methods from scientific articles.
The system utilized several ontologies for grounding, including Uberon (anatomy), ADO (Alzheimer’s Disease Ontology), and the Biolink Model.
Key Results and Insights
The researchers evaluated the performance of Claude 3.7 Sonnet, GPT-4o-mini, and DeepSeek V3 within the STRUCTSENSE framework.
- Performance Leadership: Claude 3.7 Sonnet emerged as the top performer, achieving a perfect 1.00 F1 score on deterministic schema extraction and demonstrating the highest semantic diversity in Named Entity Recognition (NER) tasks.
- The Calibration Gap: A significant finding was the "confidence mismatch" in smaller models. GPT-4o-mini often reported maximal confidence even when its output was incomplete, highlighting a dangerous calibration gap in current LLMs.
- Human-in-the-Loop (HIL) Impact: HIL feedback served as a powerful equalizer. For instance, DeepSeek’s ontology alignment accuracy improved from 40% to nearly 87% with human guidance. While HIL increases operational costs (up to 70% for Claude), it significantly improves the auditability and reliability of the data.
- Semantic Fidelity: The authors argue that traditional "exact match" metrics are often too restrictive. Models frequently extract semantically correct concepts that fail string matching but remain scientifically accurate.
Conclusion and Future Work
While the agentic overhead and human intervention make STRUCTSENSE more resource-intensive than zero-shot prompting, the gains in reliability and grounding are essential for scientific applications. Future work will focus on developing more scalable, adaptive HIL frameworks, creating evaluation metrics that reward semantic plausibility, and improving the self-calibration of LLMs so their internal "Judge" scores more accurately reflect actual performance.
Summary updated