PLLaMa: An Open-source Large Language Model for Plant Science

Xianjun Yang, Junfeng Gao, Wenxin Xue, Erik AlexanderssonarXiv 2024 · DOI · arXiv

Abstract

General-purpose large language models (LLMs) often lack the specialized knowledge required for advanced scientific research. This paper introduces PLLaMa, an open-source LLM specifically optimized for the plant science community. By extending the pretraining of LLaMa-2 on a massive corpus of 1.67 million domain-specific academic articles and performing targeted instruction finetuning, we bridge the expertise gap in agricultural and botanical sciences. PLLaMa demonstrates strong performance on technical benchmarks and provides a transparent, auditable tool for the scientific community.

Introduction

Foundational models like GPT-4 exhibit remarkable general reasoning but frequently struggle with the nuances of highly specialized fields. In plant science—a discipline vital for global food security and climate change mitigation—researchers require models that accurately interpret complex phenomena such as turgor pressure, genomic selection, and plant pathology. General web-crawl data often lacks the depth found in peer-reviewed literature. Furthermore, the "black box" nature of proprietary models limits their utility in open scientific inquiry. PLLaMa addresses these challenges by providing a high-performing, open-source alternative tailored to the specific needs of plant scientists.

Methodology

The development of PLLaMa (available in 7B and 13B parameter versions) utilized a two-stage adaptation pipeline designed to transform a general-purpose model into a domain expert.

Continued Pretraining

To facilitate knowledge acquisition, the base LLaMa-2 model was trained on a filtered subset of the Semantic Scholar Open Research Corpus (S2ORC). This subset comprised approximately 1.67 million articles extracted from 750 distinct plant science journals. To mitigate "catastrophic forgetting," where the model loses general linguistic proficiency while learning new facts, the domain-specific data was augmented with a 10% sample of general-purpose text from the RedPajama dataset.

Instruction Finetuning

To align the model for assistant-like interactions, supervised finetuning was performed. This stage utilized 1,030 high-quality instructions from the LIMA dataset, supplemented by custom-developed instructions focused on plant science topics. The training process was computationally intensive, employing eight A100 GPUs and leveraging Flash-Attention and Fully Sharded Data Parallel (FSDP) for efficiency.

Overview of the PLLaMa development pipeline, including continued pretraining on S2ORC data and instruction finetuning on customized datasets.

Results and Evaluation

PLLaMa's performance was assessed using both qualitative analysis and a newly developed quantitative benchmark. An international panel of plant scientists and breeders curated a 10-question multiple-choice quiz covering advanced topics in plant breeding and genetics.

PLLaMa achieved an accuracy of approximately 60% on this specialized benchmark. While seemingly modest, this score reflects a high level of technical competence on an exam designed for human experts. Qualitatively, the model successfully explained complex biological processes, such as endocytosis against turgor pressure and the mechanisms of cytokinesis, with high technical precision.

The 10-question plant science benchmark used to evaluate PLLaMa, showing questions, choices, and model responses.

Discussion and Future Work

The success of PLLaMa underscores the necessity of domain-specific pretraining for specialized scientific applications. Instruction tuning alone is insufficient for capturing the underlying principles of a field; models must "read" the primary literature to gain true expertise. By releasing PLLaMa as an open-source tool, we enable the community to further specialize the model for sub-fields like viticulture or forest ecology without the massive cost of initial pretraining.

Future work will focus on expanding the evaluation suite to include a more diverse range of plant science topics and increasing the variety of instruction-tuning data. Additionally, the researchers identified opportunities to better utilize the extensive corpus by optimizing data windowing in future training iterations to further improve performance.

Summary updated

BibTeX
@misc{yang2024pllama,
  author = {Xianjun Yang and Junfeng Gao and Wenxin Xue and Erik Alexandersson},
  title = {{PLLaMa: An Open-source Large Language Model for Plant Science}},
  year = {2024},
  eprint = {2401.01600},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.01600},
}
Figure