A Multi-Agent Human-LLM Collaborative Framework for Closed-Loop Scientific Literature Summarization

Maxwell J. Jacobson, Daniel Xie, Jackson Shen, Adil Wazeer, Guang Lin, Xiao-Ying Yu, Haiyan Wang, Xinghang Zhang, Yexiang XuearXiv 2026 · arXiv

TL;DR

The Elhuyar Framework is a multi-agent, human-in-the-loop system that transforms fragmented scientific literature into structured data and predictive mathematical models. By combining Large Language Models (LLMs) with iterative consensus scoring and human verification, it successfully uncovers hidden quantitative trends—such as the exponential growth of helium bubbles in nuclear reactor materials—that are difficult to synthesize manually.

What Problem Does This Paper Address?

Scientific progress is often bottlenecked by the sheer volume of fragmented literature. For researchers in fields like materials science, understanding a phenomenon often requires manually extracting data points from dozens of papers, each using different formats, units, and experimental conditions.

While current AI tools for Scientific Document Understanding (SDU) can summarize text or answer basic questions, they struggle with the "deep" insights required for discovery. They are prone to hallucinations (fabricating data) and lack the structured, multi-step reasoning needed to go from raw text to a fitted mathematical model. The authors aim to bridge this gap by creating a "closed-loop" system where AI handles the heavy lifting of extraction and modeling while humans provide high-level guidance and quality control.

Methods

The Elhuyar Framework (illustrated in Figure 2) distributes the literature review process across specialized agents:

  1. Yes/No Filtering Agent: Instead of asking the LLM to extract data immediately, this agent first asks high-level questions (e.g., "Does this paper report pure tungsten samples?") to filter the corpus. This reduces hallucination risks associated with complex extraction.
  2. Extractor Agent: This agent pulls specific independent, dependent, and control variables from the filtered papers based on a user-defined Data Definition.
  3. Iterative Consensus Scoring (ICS): To ensure reliability, the system runs the extraction multiple times ($k=10$). Data points that appear consistently across runs receive high confidence scores, while inconsistent ones are flagged for human "Inspectors."
  4. Model Selection & Fit Agents: A specialized agent selects appropriate mathematical forms (e.g., linear vs. exponential) based on the user's query. A non-LLM structured AI agent then performs standard numerical optimization (least squares regression) to fit the extracted data to these models.
  5. Report Agent: Finally, the system generates a comprehensive report including 3D visualizations, fitted equations, and a textual summary grounded strictly in the extracted data.
Figure 2: Pipeline of the Elhuyar system. It involves two human roles: the scientist, who provides literature, asks a scientific query, and defines relevant data, and the inspector, who verifies low-confidence extracted data. The extraction path begins when the scientist queries the system, triggering a yes/no filtering agent that selects relevant documents, which are then processed by an extractor agent to extract data points multiple times. Iterative consensus scoring calculates confidence to reduce hallucina- tions and misreads, ultimately producing a filtered dataset. In the modeling path, a model selection agent chooses models to compare, and a model fit+eval agent fits the filtered data to these models, generating equations and evaluations. The response path consists of a single response agent that compiles a report with visualizations, equations, and a text summary answering the scientist’s query. The final results allow the scientist to refine their question or expand the analysis with adjustments.

Datasets

The authors evaluated Elhuyar using two distinct approaches:

  • Synthetic Evaluation: A corpus of 250 LLM-generated "mini-papers" about fictional materials like "drakorium." This allowed the team to test the system against a known "ground truth" relationship between forging temperature, tempering time, and hardness.
  • Materials Science Pilot: A real-world corpus of 64 high-tier journal papers focusing on helium-ion irradiation in tungsten, a critical area for fusion reactor design.

Key Insights and Ideas

  • Reliability through Redundancy: The "aha moment" here is the Iterative Consensus Scoring. By treating LLM extraction as a probabilistic process and requiring consensus, the system effectively filters out the "noise" of hallucinations without requiring a human to read every single paper.
  • LLMs for Logic, Math for Math: The framework wisely delegates numerical fitting to traditional optimization algorithms rather than asking the LLM to "guess" a trend. The LLM's role is restricted to identifying the type of model to use and interpreting the final results.
  • Closing the Loop: The system is designed for iteration. A scientist can view the initial report and then refine their "Scientific Query" or "Data Definition" to dig deeper, effectively using the AI as a high-powered research assistant.

Key Results

In the synthetic "drakorium" test, the framework achieved 100% accuracy in extracting correct information and filtering out irrelevant papers. The fitted models reached an average $R^2$ of 0.927 against the noisy synthetic data.

In the real-world pilot, Elhuyar analyzed the complex relationship between irradiation dose, temperature, and helium bubble size (see the microscopic context in Figure 3). The system successfully identified that an exponential model ($R^2 = 0.695$) provided a significantly better fit than a linear one ($R^2 = 0.503$). This quantitative finding aligns with theoretical predictions of nonlinear helium clustering, demonstrating that the tool can clarify microstructural trends that are obscured when looking at individual, scattered studies.

Comparison of linear (A) vs. exponential (B) models fitted to extracted tungsten data. The exponential surface shows a much closer fit to the experimental data points.

Limitations and Future Work

Despite its success, the authors note several hurdles:

  • Visual Data Gap: The current system only processes text and tables. It cannot yet extract data from figures or complex charts, which are vital in many scientific papers.
  • Unit Conversion: While simple temperature conversions are automated, complex unit transformations (like converting ion fluence to displacements per atom) still require papers to report data in a specific standard or require human intervention.
  • Human Bottleneck: While the workload is reduced, human "Inspectors" are still necessary to verify mid-confidence data points to maintain the "closed-loop" integrity.

Future work aims to create a more modular codebase, allowing researchers in any field to define their own variables and run the pipeline on their own specific document corpora.

Summary updated

Abstract

Scientific discovery is slowed by fragmented literature that requires excessive human effort to gather, analyze, and understand. AI tools, including autonomous summarization and question answering, have been developed to aid in understanding scientific literature. However, these tools lack the structured, multi-step approach necessary for extracting deep insights from scientific literature. Large Language Models (LLMs) offer new possibilities for literature analysis, but remain unreliable due to hallucinations and incomplete extraction. We introduce Elhuyar, a multi-agent, human-in-the-loop system that integrates LLMs, structured AI, and human scientists to extract, analyze, and iteratively refine insights from scientific literature. The framework distributes tasks among specialized agents for filtering papers, extracting data, fitting models, and summarizing findings, with human oversight ensuring reliability. The system generates structured reports with extracted data, visualizations, model equations, and text summaries, enabling deeper inquiry through iterative refinement. Deployed in materials science, it analyzed literature on tungsten under helium-ion irradiation, showing experimentally correlated exponential helium bubble growth with irradiation dose and temperature, offering insight for plasma-facing materials (PFMs) in fusion reactors. This demonstrates how AI-assisted literature review can uncover scientific patterns and accelerate discovery.
BibTeX
@misc{jacobson2026multi,
  author = {Maxwell J. Jacobson and Daniel Xie and Jackson Shen and Adil Wazeer and Guang Lin and Xiao-Ying Yu and Haiyan Wang and Xinghang Zhang and Yexiang Xue},
  title = {{A Multi-Agent Human-LLM Collaborative Framework for Closed-Loop Scientific Literature Summarization}},
  year = {2026},
  eprint = {2604.01452},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.01452},
}
Figure