MaterialBENCH: Evaluating College-Level Materials Science Problem-Solving Abilities of Large Language Models

Michiko Yoshitake, Yuta Suzuki, Ryo Igarashi, Yoshitaka Ushiku, Keisuke Nagato

Abstract

As Large Language Models (LLMs) increasingly assist in scientific research, evaluating their domain-specific reasoning becomes critical. This paper introduces MaterialBENCH, a benchmark designed to assess the proficiency of LLMs in college-level materials science. By evaluating models like GPT-4, GPT-3.5, and Bard across 164 problems, we identify a significant performance gap between generations of models and highlight persistent challenges in spatial reasoning, multi-step calculations, and the necessity of Chain-of-Thought (CoT) processing.

Introduction

While Large Language Models (LLMs) have demonstrated remarkable general-purpose reasoning, their utility in specialized scientific domains remains inconsistent. Existing benchmarks often focus on broad knowledge (MMLU) or specific chemical safety and organic chemistry (ChemBench). Materials science, however, presents a unique set of challenges: it requires a deep understanding of the "state of aggregation"—how atomic and molecular arrangements dictate macroscopic mechanical, electrical, and magnetic properties. MaterialBENCH was developed to determine whether current AI models can authentically reason through materials engineering problems or if they merely retrieve memorized textbook definitions.

The MaterialBENCH Dataset

MaterialBENCH consists of 164 problems meticulously curated through a "human-in-the-loop" methodology. A materials science expert collected and verified problems from two foundational university textbooks: Materials Science and Engineering: An Introduction (Callister & Rethwisch) and Understanding Materials Science (Hummel).

The dataset covers a wide spectrum of topics, including:

  • Atomic bonding and crystallography
  • Diffusion mechanisms
  • Mechanical properties and strength
  • Corrosion and degradation
  • Magnetic and electrical properties

To isolate linguistic and mathematical reasoning from visual processing, the authors intentionally excluded problems involving phase diagrams or complex micrographs, as these would require multimodal vision capabilities beyond the scope of this specific study.

Methodology

The study evaluated three prominent models: ChatGPT-3.5, ChatGPT-4, and Bard (now Gemini). Each model was tested using two distinct formats:

  1. Free-response: Models provided full, step-by-step written solutions.
  2. Multiple-choice: Models selected from four options, including "distractor" answers modeled after common student misconceptions.

To ensure statistical stability, the researchers utilized the OpenAI API to implement a "majority voting" strategy. Each question was prompted 10 times, and the most frequent answer was selected as the final model response. This approach allowed the team to compare the stability of model reasoning against individual attempt accuracy.

Key Findings and Analysis

The Dominance of GPT-4

The results revealed a stark divide in performance. In free-response tasks, GPT-4 achieved an accuracy of 0.64, more than doubling the performance of ChatGPT-3.5 (0.28) and Bard (0.30). When utilizing majority voting in a multiple-choice format, GPT-4’s accuracy peaked at 0.81, demonstrating its superior capacity for handling specialized technical content.

The Impact of Prompt Constraints on Reasoning

A critical insight emerged regarding "system messages." When models were constrained by strict instructions to provide only the final answer (e.g., "Answer with one correct choice"), accuracy plummeted. For GPT-4, multiple-choice accuracy dropped from 0.81 to 0.54. This suggests that suppressing the model’s "Chain of Thought" (CoT) prevents it from processing the intermediate logical steps required for complex materials science problems, leading to impulsive and incorrect conclusions.

Computational and Semantic Hurdles

Despite GPT-4's success, mathematical precision remains a bottleneck. Models frequently struggled with calculations involving factorials and exponents. GPT-4 mitigated this somewhat by leveraging Python-based plugins for heavy computation, whereas other models relied on internal (and often flawed) arithmetic.

Furthermore, the study identified "semantic misunderstandings" where models lacked physical intuition. For instance, when calculating the number of atoms in a Hexagonal Close-Packed (HCP) unit cell, models often defaulted to calculating for a general hexagonal prism, failing to apply the specific geometric constraints of a "unit cell" in a materials context.

Conclusion

MaterialBENCH highlights both the progress and the "reasoning ceiling" of current LLMs. While GPT-4 shows promise as a tool for materials scientists, the existence of problems that no model could solve—regardless of prompting—indicates that multi-stage reasoning in physical sciences is still an evolving frontier. Future iterations of this work will likely incorporate multimodal evaluations to include phase diagrams and expand the dataset to include research-level complexities beyond introductory textbooks.

Summary updated

Abstract

A college-level benchmark dataset for large language models (LLMs) in the materials science field, MaterialBENCH, is constructed. This dataset consists of problem-answer pairs, based on university textbooks. There are two types of problems: one is the free-response answer type, and the other is the multiple-choice type. Multiple-choice problems are constructed by adding three incorrect answers as choices to a correct answer, so that LLMs can choose one of the four as a response. Most of the problems for free-response answer and multiple-choice types overlap except for the format of the answers. We also conduct experiments using the MaterialBENCH on LLMs, including ChatGPT-3.5, ChatGPT-4, Bard (at the time of the experiments), and GPT-3.5 and GPT-4 with the OpenAI API. The differences and similarities in the performance of LLMs measured by the MaterialBENCH are analyzed and discussed. Performance differences between the free-response type and multiple-choice type in the same models and the influence of using system massages on multiple-choice problems are also studied. We anticipate that MaterialBENCH will encourage further developments of LLMs in reasoning abilities to solve more complicated problems and eventually contribute to materials research and discovery.
BibTeX
@misc{yoshitake2024materialbench,
  author = {Michiko Yoshitake and Yuta Suzuki and Ryo Igarashi and Yoshitaka Ushiku and Keisuke Nagato},
  title = {{MaterialBENCH: Evaluating College-Level Materials Science Problem-Solving Abilities of Large Language Models}},
  year = {2024},
  eprint = {2409.03161},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.03161},
}
Figure