Abstract
We propose an instruction-based process for trustworthy data curation in materials science (MatSci-Instruct), which we then apply to finetune a LLaMa-based language model targeted for materials science (HoneyBee). MatSci-Instruct helps alleviate the scarcity of relevant, high-quality materials science textual data available in the open literature, and HoneyBee is the first billion-parameter language model specialized to materials science. In MatSci-Instruct we improve the trustworthiness of generated data by prompting multiple commercially available large language models for generation with an Instructor module (e.g. Chat-GPT) and verification from an independent Verifier module (e.g. Claude). Using MatSci-Instruct, we construct a dataset of multiple tasks and measure the quality of our dataset along multiple dimensions, including accuracy against known facts, relevance to materials science, as well as completeness and reasonableness of the data. Moreover, we iteratively generate more targeted instructions and instruction-data in a finetuning-evaluation-feedback loop leading to progressively better performance for our finetuned HoneyBee models. Our evaluation on the MatSci-NLP benchmark shows HoneyBee's outperformance of existing language models on materials science tasks and iterative improvement in successive stages of instruction-data refinement. We study the quality of HoneyBee's language modeling through automatic evaluation and analyze case studies to further understand the model's capabilities and limitations. Our code and relevant datasets are publicly available.
Introduction
Materials science is a cornerstone of modern innovation, yet much of its critical knowledge is locked away in millions of scientific papers and technical reports. While general-purpose large language models (LLMs) like GPT-4 are impressive, they often struggle with the nuances of specialized scientific domains. Conversely, existing domain-specific models like MatBERT lack the generative power of modern architectures. The primary bottleneck has been a "data scarcity" problem: the lack of high-quality, instruction-based textual data available to train massive models specifically for materials science.
The MatSci-Instruct Framework
To address this, the researchers developed MatSci-Instruct, a multi-agent framework designed to curate trustworthy training data by leveraging the strengths of multiple frontier LLMs.
The framework employs a three-part system:
- The Instructor (ChatGPT): Generates domain-specific instructions and responses based on materials science topics or arXiv snippets.
- The Verifier (Claude): Acts as a quality controller, scoring data on accuracy, relevance, completeness, and reasonableness. Only high-scoring data is retained.
- The Evaluator (GPT-4): Assesses the final output of the trained model to identify weaknesses.
This architecture enables Progressive Instruction Finetuning. Instead of a single training pass, the model is refined across multiple stages. If the Evaluator identifies a weakness, the Instructor is prompted to generate more data for that specific topic, leading to iterative performance gains.
Model Development and Datasets
The resulting model, HoneyBee, was developed in 7B and 13B parameter versions using Low-Rank Adaptation (LoRA). The training utilized the MatSci-Instruct dataset, which contains approximately 52,000 instructions covering 20 materials science subtopics—including graphene, semiconductors, and biomaterials—and 20 distinct NLP tasks.
Key Results and Evaluation
HoneyBee-13b (Stage 3) achieved an accuracy of 98.11% on MatSci-Instruct tasks, notably outperforming its own "teacher" model, ChatGPT (92.55%). On the MatSci-NLP benchmark, HoneyBee surpassed existing specialized models like MatBERT and MatSciBERT across nearly all tasks, including relation extraction and paragraph classification.
A key finding was the high alignment between automated metrics and human expertise. Human evaluations conducted by materials science graduate students showed a correlation of up to 0.8 with the scores provided by Claude and GPT-4, validating the "trustworthy" curation pipeline as a reliable method for domain-specific distillation.
Conclusion
HoneyBee demonstrates that domain-specific distillation is a viable path for creating powerful, deployable scientific tools that are significantly more efficient than general-purpose giants. While challenges remain in complex physical reasoning and synthesis recipe generation, the progressive finetuning approach offers a robust template for specializing LLMs in other scientific disciplines.
Summary updated