Abstract
The reproducibility crisis in biological research is frequently attributed to the lack of precise, executable experimental protocols. While large language models (LLMs) have demonstrated significant capabilities in scientific text generation, they often fail to maintain the operational logic necessary for laboratory workflows, leading to "hallucinated" steps or incorrect sequencing. We introduce Thoth, a specialized LLM designed to generate high-fidelity, executable protocols. Thoth utilizes a "Sketch-and-Fill" reasoning paradigm and is optimized via a novel Structured COmponent-based REward (SCORE) mechanism. This mechanism shifts the focus from linguistic similarity to logical and semantic accuracy. Trained on SciRecipe, a comprehensive dataset of 12,000 structured protocols, Thoth significantly outperforms general-purpose models, including GPT-4o, across multiple scientific benchmarks.
Introduction: The Reproducibility Crisis in Protocol Generation
In the life sciences, the ability to replicate an experiment is the cornerstone of scientific validity. However, many published protocols are vague or poorly documented, leading to a widespread "reproducibility crisis." While LLMs are increasingly used as research assistants, they struggle to act as "operational architects." Standard models often produce instructions that are logically flawed—for instance, suggesting a sample be centrifuged before a lysis buffer is added.
Traditional evaluation metrics like BLEU or ROUGE exacerbate this problem by rewarding text that looks like a reference, regardless of whether the underlying logic is sound. To address this, we propose a shift toward evaluating the "executability" of protocols, ensuring that actions, objects, and parameters are correctly sequenced and specified.
Methodology: Sketch-and-Fill and the SCORE Mechanism
Thoth addresses the limitations of standard LLMs through a structured reasoning approach called the Sketch-and-Fill paradigm. Instead of generating a protocol as a single block of text, the model follows a multi-stage process:
<think>: The model decomposes the experimental goal into sub-goals.<key>: It generates a sequence of atomic, machine-readable steps (Action-Object-Parameter triplets).<orc>: These steps are expanded into fluent natural language.<note>: Safety precautions and laboratory tips are added.
To guide the model's learning, we developed the SCORE (Structured COmponent-based REward) mechanism. Used during Reinforcement Learning (RL), SCORE evaluates three critical dimensions: Step Granularity (ensuring appropriate detail), Action Ordering (penalizing logical deviations), and Semantic Fidelity (verifying specific parameters like volumes and temperatures).
Data and Training
The model was trained on SciRecipe, a newly curated dataset containing over 12,000 structured protocols across 27 biological subfields. Training proceeded in three stages: initial exposure to large-scale protocol texts, Supervised Fine-Tuning (SFT) on the Sketch-and-Fill format, and finally Reinforcement Learning using the Group Relative Policy Optimization (GRPO) algorithm, guided by SCORE rewards.
Experimental Results
Thoth (8B) was evaluated against a suite of general-purpose and reasoning-focused models. On the SciRecipe-Eval benchmark, Thoth consistently outperformed proprietary models like ChatGPT-4o, particularly in metrics measuring logical sequencing and semantic alignment.
Beyond protocol-specific tasks, Thoth demonstrated strong generalization across broader scientific benchmarks, including Humanity’s Last Exam (HLE), LAB-Bench, and PubMedQA. As shown in the results, Thoth achieves state-of-the-art accuracy, suggesting that mastering the logic of procedures enhances general scientific reasoning capabilities.
Conclusion
Thoth represents a significant step toward automated, reproducible science. By prioritizing the structural logic of experimental workflows over linguistic fluency, the model provides a reliable tool for generating executable lab protocols. Future work will focus on expanding the dataset to include more specialized techniques and refining the reward mechanism to accommodate scientifically valid variations in experimental design.
Summary updated