OPENXRD: A Comprehensive Benchmark and Enhancement Framework for LLM/MLLM XRD Question Answering

Ali Vosoughi, Ayoub Shahnazari, Yufeng Xi, Zeliang Zhang, Griffin Hess, Chenliang Xu, Niaz AbdolrahimarXiv 2025 · DOI · arXiv

Abstract

As Large Language Models (LLMs) increasingly penetrate scientific domains, their ability to handle specialized tasks like X-ray diffraction (XRD) remains under-explored. This paper introduces OPENXRD, a benchmarking framework designed to evaluate LLMs on complex crystallographic reasoning. By testing 74 state-of-the-art models, we demonstrate that while frontier models approach a performance ceiling, mid-sized models (7B–70B parameters) derive the greatest benefit from expert-curated context, effectively bridging the gap to larger counterparts. Our findings highlight the critical role of "context assimilation" and the persistent challenge of symbolic mathematical reasoning in scientific AI.

Introduction

X-ray diffraction (XRD) is the foundational technique for determining the atomic structure of materials. Despite its importance, mastering XRD requires deep expertise in physics, mathematics, and crystallography. While traditional deep learning models excel at pattern recognition in raw XRD data, they lack the interpretability and reasoning capabilities required for scientific discovery.

Large Language Models (LLMs) offer a potential path toward interpretable scientific assistants. However, their tendency to hallucinate in niche domains and the lack of specialized benchmarks hinder their deployment. OPENXRD addresses this by isolating "context assimilation"—the ability of a model to integrate provided scientific knowledge to solve domain-specific problems—rather than relying solely on pre-trained internal knowledge.

Methodology

The OPENXRD framework employs a dual-mode evaluation strategy to measure model performance and the impact of external knowledge:

  1. Closed-Book Mode: Models answer questions using only their internal parameters.
  2. Open-Book Mode: Models are provided with concise, domain-relevant reference passages to simulate a Retrieval-Augmented Generation (RAG) environment.
The OPENXRD evaluation workflow, illustrating the fusion of questions with supporting textual material for LLM inference.

To evaluate the importance of data quality, the researchers compared reference passages generated by GPT-4.5 against those meticulously refined by crystallography experts. This allowed for the measurement of the "expert refinement differential," testing whether human-level precision in scientific terminology acts as a performance multiplier.

The OPENXRD Dataset

The study is built upon a core dataset of 217 expert-curated multiple-choice questions. These questions span the full spectrum of crystallography, from fundamental concepts like Bragg’s Law to advanced topics such as space-group interpretation and scattering physics. The dataset includes detailed subtask labels and token-matched reference passages (AI-generated vs. Expert-refined) to ensure a controlled comparison of context utility.

Key Results and Analysis

The evaluation of 74 models revealed distinct performance tiers based on model size and reasoning capacity. Frontier models, such as GPT-5 and Grok-4-fast, demonstrated exceptional internal knowledge, achieving closed-book accuracy scores exceeding 96%.

Closed-book performance across model size categories, showing a clear correlation between parameter count and baseline accuracy.

The "Goldilocks" Zone for Context

A primary finding is that mid-sized models (7B–70B parameters) are the primary beneficiaries of external context. These models possess sufficient reasoning "horsepower" to process external help but have enough internal knowledge gaps that the context provides a significant boost. For instance, LLaVA-v1.6-34B improved from 66.8% to 78.3% when provided with expert materials, rivaling much larger models.

Contextual Interference and Math Limitations

Conversely, providing context to the most advanced "frontier" models occasionally resulted in "conceptual interference," where the provided text conflicted with their deep internal knowledge, leading to a slight decrease in accuracy. Furthermore, a significant "Math Wall" was observed: nearly all models, regardless of size or context, failed complex symbolic math derivations (e.g., structure factor calculations), suggesting that linguistic context is insufficient for formal algebraic problem-solving.

Conclusion

OPENXRD provides a rigorous diagnostic tool for evaluating AI in the physical sciences. While expert-refined context significantly enhances the performance of mid-sized models, the persistent failure in symbolic mathematics indicates that future scientific LLMs must be integrated with symbolic computation engines. The OPENXRD framework is open-source, serving as a foundation for the next generation of multimodal scientific assistants capable of interpreting both text and complex diffraction imagery.

Summary updated

Abstract

We introduce OPENXRD, a comprehensive benchmarking framework for evaluating large language models (LLMs) and multimodal LLMs (MLLMs) in crystallography question answering. The framework measures context assimilation, or how models use fixed, domain-specific supporting information during inference. The framework includes 217 expert-curated X-ray diffraction (XRD) questions covering fundamental to advanced crystallographic concepts, each evaluated under closed-book (without context) and open-book (with context) conditions, where the latter includes concise reference passages generated by GPT-4.5 and refined by crystallography experts. We benchmark 74 state-of-the-art LLMs and MLLMs, including GPT-4, GPT-5, O-series, LLaVA, LLaMA, QWEN, Mistral, and Gemini families, to quantify how different architectures and scales assimilate external knowledge. Results show that mid-sized models (7B--70B parameters) gain the most from contextual materials, while very large models often show saturation or interference and the largest relative gains appear in small and mid-sized models. Expert-reviewed materials provide significantly higher improvements than AI-generated ones even when token counts are matched, confirming that content quality, not quantity, drives performance. OPENXRD offers a reproducible diagnostic benchmark for assessing reasoning, knowledge integration, and guidance sensitivity in scientific domains, and provides a foundation for future multimodal and retrieval-augmented crystallography systems.
BibTeX
@misc{vosoughi2025openxrd,
  author = {Ali Vosoughi and Ayoub Shahnazari and Yufeng Xi and Zeliang Zhang and Griffin Hess and Chenliang Xu and Niaz Abdolrahim},
  title = {{OPENXRD: A Comprehensive Benchmark and Enhancement Framework for LLM/MLLM XRD Question Answering}},
  year = {2025},
  eprint = {2507.09155},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.09155},
}
Figure