HalluMat: Detecting Hallucinations in LLM-Generated Materials Science Content Through Multi-Stage Verification

Bhanu Prakash Vangala, Sajid Mahmud, Pawan Neupane, Joel Selvaraj, Jianlin ChengarXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) are increasingly applied to materials science to accelerate discovery, yet they frequently produce "hallucinations"—plausible but scientifically inaccurate information. We present HalluMat, a comprehensive framework and dataset designed to detect and mitigate these hallucinations. HalluMat utilizes a multi-stage verification pipeline, incorporating intrinsic evaluation, extrinsic retrieval-augmented checks, and knowledge graph analysis. Furthermore, we introduce the Paraphrased Hallucination Consistency Score (PHCS) to quantify model reliability across semantically equivalent queries. Our results demonstrate a 30% reduction in hallucination rates and an 82.2% detection accuracy on the novel HalluMatData benchmark, offering a robust path toward trustworthy AI in scientific domains.

Introduction

Large Language Models (LLMs) like GPT-4 and LLaMA have shown promise in predicting material properties and suggesting synthesis methods. However, in a scientific context, hallucinations are a significant liability. Proposing physically impossible crystal structures can lead researchers toward expensive and time-consuming dead ends. Existing detection methods often fail in specialized fields because they rely on general-purpose databases like Wikipedia, which lack the technical depth of materials science literature. Moreover, generic methods do not account for "semantic drift"—the inconsistency that occurs when the same scientific question is phrased differently.

Methodology

The HalluMatDetector Pipeline

The HalluMatDetector uses a multi-stage "verify-then-check" approach to ensure factual integrity:

  1. Intrinsic Evaluation: The model performs self-checks by analyzing self-consistency, confidence variance, and token entropy. High entropy often correlates with factual inconsistencies.
  2. Extrinsic Verification: If internal checks are inconclusive, the system triggers a fallback to external sources, retrieving snippets from Google Scholar and Wikipedia using FAISS for dense retrieval and BM25 for keyword reranking.
  3. Knowledge Graph Analysis: The system decomposes AI responses into "fact fragments" and maps them onto a graph. Factual responses typically form tight, logical clusters, while hallucinations appear as fragmented, disconnected nodes.
2D Knowledge Graph Visualization of Fact Fragments

The PHCS Metric

To address the fragility of model responses to phrasing, we introduce the Paraphrased Hallucination Consistency Score (PHCS). This metric measures how much an AI’s truthfulness fluctuates when a query is rephrased. A rising PHCS score indicates that the model is unreliable for a specific topic, even if it occasionally provides a correct answer.

HalluMatData Dataset

The study introduces HalluMatData, the first curated benchmark dataset specifically for evaluating hallucinations in materials science. It contains 2,629 unique scientific queries and 640 paraphrased versions. The dataset includes 3,269 AI-generated responses categorized as "Low," "Medium," or "High" hallucination levels, verified against authoritative sources like the Materials Project.

Key Results

The HalluMatDetector pipeline reduced hallucination rates by 30% compared to standard, unverified LLM outputs. The framework achieved an overall detection accuracy of 82.2% on the HalluMatData benchmark.

Confusion Matrix: Computed vs Recomputed Hallucination Levels

As shown in the confusion matrix, the system is particularly effective at reassessing "Medium" hallucination cases. Many instances initially flagged as medium-level hallucinations were correctly reclassified as high-level hallucinations upon deeper verification, significantly reducing the risk of false positives in scientific data.

Conclusion and Future Work

HalluMat represents a major step toward reliable AI in materials science. While the system currently achieves moderate precision and recall, future iterations will explore domain-specific embeddings (e.g., MatSciBERT) to capture more subtle technical errors. Additionally, incorporating Reinforcement Learning from Human Feedback (RLHF) during the training phase could help penalize inconsistent responses before they are generated, further hardening LLMs against scientific hallucinations.

Summary updated

Abstract

Artificial Intelligence (AI), particularly Large Language Models (LLMs), is transforming scientific discovery, enabling rapid knowledge generation and hypothesis formulation. However, a critical challenge is hallucination, where LLMs generate factually incorrect or misleading information, compromising research integrity. To address this, we introduce HalluMatData, a benchmark dataset for evaluating hallucination detection methods, factual consistency, and response robustness in AI-generated materials science content. Alongside this, we propose HalluMatDetector, a multi-stage hallucination detection framework that integrates intrinsic verification, multi-source retrieval, contradiction graph analysis, and metric-based assessment to detect and mitigate LLM hallucinations. Our findings reveal that hallucination levels vary significantly across materials science subdomains, with high-entropy queries exhibiting greater factual inconsistencies. By utilizing HalluMatDetector verification pipeline, we reduce hallucination rates by 30% compared to standard LLM outputs. Furthermore, we introduce the Paraphrased Hallucination Consistency Score (PHCS) to quantify inconsistencies in LLM responses across semantically equivalent queries, offering deeper insights into model reliability.
BibTeX
@misc{vangala2025hallumat,
  author = {Bhanu Prakash Vangala and Sajid Mahmud and Pawan Neupane and Joel Selvaraj and Jianlin Cheng},
  title = {{HalluMat: Detecting Hallucinations in LLM-Generated Materials Science Content Through Multi-Stage Verification}},
  year = {2025},
  eprint = {2512.22396},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.22396},
}
Figure