Abstract
As Large Language Models (LLMs) increasingly penetrate scientific domains, their ability to handle specialized tasks like X-ray diffraction (XRD) remains under-explored. This paper introduces OPENXRD, a benchmarking framework designed to evaluate LLMs on complex crystallographic reasoning. By testing 74 state-of-the-art models, we demonstrate that while frontier models approach a performance ceiling, mid-sized models (7B–70B parameters) derive the greatest benefit from expert-curated context, effectively bridging the gap to larger counterparts. Our findings highlight the critical role of "context assimilation" and the persistent challenge of symbolic mathematical reasoning in scientific AI.
Introduction
X-ray diffraction (XRD) is the foundational technique for determining the atomic structure of materials. Despite its importance, mastering XRD requires deep expertise in physics, mathematics, and crystallography. While traditional deep learning models excel at pattern recognition in raw XRD data, they lack the interpretability and reasoning capabilities required for scientific discovery.
Large Language Models (LLMs) offer a potential path toward interpretable scientific assistants. However, their tendency to hallucinate in niche domains and the lack of specialized benchmarks hinder their deployment. OPENXRD addresses this by isolating "context assimilation"—the ability of a model to integrate provided scientific knowledge to solve domain-specific problems—rather than relying solely on pre-trained internal knowledge.
Methodology
The OPENXRD framework employs a dual-mode evaluation strategy to measure model performance and the impact of external knowledge:
- Closed-Book Mode: Models answer questions using only their internal parameters.
- Open-Book Mode: Models are provided with concise, domain-relevant reference passages to simulate a Retrieval-Augmented Generation (RAG) environment.
To evaluate the importance of data quality, the researchers compared reference passages generated by GPT-4.5 against those meticulously refined by crystallography experts. This allowed for the measurement of the "expert refinement differential," testing whether human-level precision in scientific terminology acts as a performance multiplier.
The OPENXRD Dataset
The study is built upon a core dataset of 217 expert-curated multiple-choice questions. These questions span the full spectrum of crystallography, from fundamental concepts like Bragg’s Law to advanced topics such as space-group interpretation and scattering physics. The dataset includes detailed subtask labels and token-matched reference passages (AI-generated vs. Expert-refined) to ensure a controlled comparison of context utility.
Key Results and Analysis
The evaluation of 74 models revealed distinct performance tiers based on model size and reasoning capacity. Frontier models, such as GPT-5 and Grok-4-fast, demonstrated exceptional internal knowledge, achieving closed-book accuracy scores exceeding 96%.
The "Goldilocks" Zone for Context
A primary finding is that mid-sized models (7B–70B parameters) are the primary beneficiaries of external context. These models possess sufficient reasoning "horsepower" to process external help but have enough internal knowledge gaps that the context provides a significant boost. For instance, LLaVA-v1.6-34B improved from 66.8% to 78.3% when provided with expert materials, rivaling much larger models.
Contextual Interference and Math Limitations
Conversely, providing context to the most advanced "frontier" models occasionally resulted in "conceptual interference," where the provided text conflicted with their deep internal knowledge, leading to a slight decrease in accuracy. Furthermore, a significant "Math Wall" was observed: nearly all models, regardless of size or context, failed complex symbolic math derivations (e.g., structure factor calculations), suggesting that linguistic context is insufficient for formal algebraic problem-solving.
Conclusion
OPENXRD provides a rigorous diagnostic tool for evaluating AI in the physical sciences. While expert-refined context significantly enhances the performance of mid-sized models, the persistent failure in symbolic mathematics indicates that future scientific LLMs must be integrated with symbolic computation engines. The OPENXRD framework is open-source, serving as a foundation for the next generation of multimodal scientific assistants capable of interpreting both text and complex diffraction imagery.
Summary updated