Benchmarking language models and agents for scientific reasoning in evolving materials laboratories

Xuyu Dong, Yixuan Li, Jianguo Huang, Feng Chen, Yiyun Zhou, Bo An, Zheng Liu, Da ShuarXiv 2026 · arXiv

TL;DR

The paper introduces MatScope, a comprehensive benchmark designed to evaluate how well large language models (LLMs) and AI agents perform "scientific reasoning" within the context of a materials laboratory. The results reveal a significant gap: while top-tier models excel at general physical explanations, they struggle with the precise quantitative estimates and the ability to maintain consistent logic as experimental evidence accumulates over time.

What Problem Does This Paper Address?

Materials discovery is rarely a single "Aha!" moment; it is a trajectory. Researchers must interpret characterization data, compare specimens across different processing branches, and update their hypotheses as new measurements (like grain size or phase fraction) come in.

Existing benchmarks for LLMs in science typically focus on static domain knowledge (Q&A) or basic tool use (calling an API). However, they don't test state-specific reasoning—the ability to distinguish between two nearly identical alloys treated at slightly different temperatures—or evolutionary adaptation, where a model must update its predictions for a final property (like tensile strength) based on intermediate structural observations. The authors argue that for AI to truly "drive" a laboratory, it must be able to handle these fine-grained, path-dependent judgements.

Methods

The authors developed a three-tiered evaluation framework:

  1. M1 (Unaided Reasoning): 200 tasks testing "zero-shot" knowledge across ten functions, such as "condition binding" (linking a response to a specific treatment) and "cross-scale interpretation" (linking atomic features to macroscopic properties).
  2. M2 (Agent-Assisted Transfer): A test of "agentic" capabilities. Models are given tools to search, read, and calculate using a curated library of ~29,000 literature works to solve problems they failed in M1.
  3. MatEvoBench-20 (ME20): The most novel component. It simulates a research pathway where evidence is released sequentially. Models must forecast a chain of hidden properties and then revise their remaining predictions as each "observation" is revealed.
The MatScope framework: from local M1 tasks to linked ME20 research pathways.

Datasets

The study utilizes a massive, deduplicated corpus of 28,872 literature works, primarily focusing on aluminum alloys and high-entropy alloys (HEAs).

  • Core200: A set of 200 expert-validated questions derived from this corpus.
  • ME20: 20 complex research trajectories involving 90 distinct "nodes" (data points) connected by scientific dependencies.

Key Insights and Ideas

  • The "Calculation vs. Estimation" Paradox: Models are incredibly good at math when you give them the numbers (92–99% success on "given-input" calculations). However, they are poor at estimating those same numbers from a description of a material state (22–31% success). They know the formulas, but they don't have a "feel" for the physical magnitudes.
  • Plausible but Wrong: Models often provide correct physical explanations (e.g., "precipitate hardening will increase strength") while simultaneously predicting a value that is off by an order of magnitude or even reversing a comparison between two samples.
  • The Instability of Evidence: In the ME20 "evolutionary" tests, new observations sometimes acted as a "double-edged sword." While some predictions improved, others that were initially accurate became less accurate as the model struggled to balance competing physical mechanisms (like solute redistribution vs. precipitation).
  • Knowledge Transfer is Brittle: Even with access to search tools (M2), models often failed to "calibrate" data from a related study to their specific target problem, often ignoring differences in cooling rates or minor composition shifts.

Key Results

The study evaluated eleven model deployments, including advanced (and perhaps internal/future-dated) versions like GPT-6 Astra and Claude Fable 5.1.

  • Low Baseline: In the unaided M1 tasks, the overall success rate was surprisingly low, ranging from only 13.5% to 19.0%.
  • Marginal Agentic Gains: Giving models search tools (M2) only provided a small boost. GPT-6 Astra improved by 1.6 percentage points, while Claude Fable 5.1 gained 4.2 points.
  • The Hidden Gap: In the ME20 pathways, models were perfect (100%) at "recovering" records that had already been released, but their accuracy for hidden future forecasts remained stuck around 50–58%, even at the very last step of the experiment.

Limitations and Future Work

The authors acknowledge that the benchmark is currently focused on specific alloy systems (Al and HEAs), which may not represent the full diversity of materials science. Furthermore, the "agent" setups were bounded by token limits and a fixed number of search turns.

The paper concludes that the next frontier for AI in science isn't just "more data," but the development of materials-aware world models. These models need to maintain internal quantitative consistency across different scales (from atoms to beams) and different points in time as an experiment evolves. MatScope provides the "measurable targets" needed to track progress toward that goal.

Summary updated

Abstract

Large language models and the multi-agent systems built around them could accelerate materials discovery by connecting scientific knowledge with experimental decisions. Existing benchmarks establish capabilities in domain knowledge, multimodal interpretation and scientific tool use. As language models increasingly support materials research workflows, their ability to distinguish closely related material states and reason accurately with quantitative evidence remains insufficiently evaluated. These abilities are essential for comparing material responses, assessing mechanisms and maintaining reliable judgements as experimental evidence accumulates. We introduce MatScope to evaluate three connected capabilities. M1 tests unaided, condition-specific reasoning through 200 tasks spanning ten research functions, using aluminium and high-entropy alloys as representative material systems. M2 tests agent-assisted transfer of related knowledge to unresolved materials questions. MatEvoBench-20 tests initial forecasts across scientific dependency chains and revisions of still-hidden responses as observations accumulate. Across eleven model deployments, including GPT-6 Astra and Claude Fable 5.1, given-input calculations and many physical explanations are handled well. Fine-grained response estimates, comparisons between similar states and quantitatively supported choices remain substantially weaker. Related literature improves selected estimates, while condition and baseline mismatches limit transfer. Progressive observations calibrate some downstream responses and destabilize earlier accurate predictions. MatScope establishes measurable targets for developing reliable scientific reasoning in future self-evolving materials laboratories and materials-aware world models.
BibTeX
@misc{dong2026benchmarking,
  author = {Xuyu Dong and Yixuan Li and Jianguo Huang and Feng Chen and Yiyun Zhou and Bo An and Zheng Liu and Da Shu},
  title = {{Benchmarking language models and agents for scientific reasoning in evolving materials laboratories}},
  year = {2026},
  eprint = {2611.12014},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.12014},
}
Figure