Validation of an LLM-based Multi-Agent Framework for Protein Engineering in Dry Lab and Wet Lab

Yiqing Shen, Zan Chen, Yungeng Liu, Yu Guang Wang, Yiqing Shen 0003BIBM 2024 · DOI

TL;DR

This paper introduces and validates TourSynbio-Agent, a novel LLM-based multi-agent framework designed to automate complex protein engineering workflows. By integrating a protein-specialized multimodal LLM with domain-specific agents, TourSynbio-Agent successfully bridges the gap between computational predictions and experimental validation, demonstrating its potential to accelerate scientific discovery in protein engineering.

What Problem Does This Paper Address?

Protein engineering, a field crucial for developing new therapeutics, enzymes, and biomaterials, has seen significant advancements with deep learning (DL) models for tasks like protein structure prediction. However, the widespread adoption of these powerful DL tools is hindered by their technical complexity, requiring specialized expertise in both protein science and DL. While Large Language Models (LLMs) have shown promise in interpreting protein-related information, they have largely been confined to analytical functions, lacking the ability to autonomously execute entire engineering workflows. This paper addresses this gap by presenting and validating TourSynbio-Agent, a framework that aims to democratize protein engineering by enabling users to interact with complex DL models through an intuitive, conversational interface.

Methods

The core of TourSynbio-Agent is its multi-agent architecture, which combines TourSynbio-7B, a protein-specialized multimodal LLM, with several domain-specific DL models. TourSynbio-7B is unique in its ability to process protein sequences directly as natural language, eliminating the need for complex external encoders. This LLM acts as the central orchestrator, interpreting user instructions and delegating tasks to specialized agents. These agents leverage established DL models for specific protein engineering tasks:

  • Mutation Prediction Agent: Utilizes the ESM-1v model to predict the functional impact of amino acid substitutions.
  • Protein Folding Agent: Employs ESMfold to generate three-dimensional protein structure predictions from sequences.
  • Protein Design Agent: Integrates the AntiFold module for inverse folding, enabling the design of novel protein sequences that conform to desired structural templates.

The framework's workflow is designed to be conversational, allowing users to specify tasks and parameters through natural language queries.

Datasets

The paper does not explicitly detail datasets used for training the TourSynbio-7B model or the individual DL agents. However, it references existing benchmarks and models:

  • ProteinLMBench [11]: Used for initial benchmarking of TourSynbio-7B's protein property analysis capabilities.
  • ESM-1v [13]: A protein language model for variant prediction.
  • ESMFold [14]: A model for end-to-end atomic-level structure prediction.
  • AntiFold [17]: A module for inverse folding and protein design.
  • IMGT standards [16]: Used for structural templates in protein design.

The wet-lab studies involved experimental validation using real protein sequences and generated variants.

Key Insights and Ideas

  • LLMs as Orchestrators for Complex Science: The paper demonstrates that LLMs can move beyond simple text generation or analysis to actively orchestrate complex scientific workflows involving multiple specialized DL models.
  • Bridging the Expertise Gap: By providing a natural language interface, TourSynbio-Agent significantly lowers the barrier to entry for using advanced protein engineering tools, making them accessible to a wider range of researchers.
  • Seamless Dry Lab to Wet Lab Integration: The framework's validation through both computational (dry lab) and experimental (wet lab) case studies highlights its practical utility and potential to accelerate the iterative cycle of design, prediction, and validation.
  • Automating Complex Tasks: The framework successfully automates tasks like mutation prediction, protein folding, and protein design, which are traditionally time-consuming and require deep expertise.

Key Results

The study presents results from five case studies:

  • Mutation Prediction: TourSynbio-Agent successfully predicted mutation effects, identifying variants like H24M with a significantly higher predicted activity score (7.84565) compared to the wild-type.
  • Protein Folding: The framework accurately predicted protein structures, achieving a pLDDT confidence score of 78.7073% for a given sequence, indicating high prediction reliability.
  • Protein Design: The framework generated structurally consistent antibody variants, with one exemplar design achieving a global score of 1.0470.
  • Wet-lab Study I (P450 Engineering): TourSynbio-Agent assisted in engineering P450 enzymes, achieving up to a 70% improvement in selectivity for steroid 19-hydroxylation. The framework's predictions showed a correlation coefficient of 0.7 with experimental measurements.
  • Wet-lab Study II (Reductase Engineering): The framework helped engineer reductase enzymes, leading to a 3.7× enhancement in catalytic conversion rate for alcohol compounds. Again, predictions showed a correlation coefficient of 0.7 with experimental results.

Limitations and Future Work

  • Limitations: The paper does not explicitly detail the training datasets for the LLM or agents, which could be a limitation for reproducibility and understanding model biases. The correlation coefficient of 0.7 in the wet-lab studies, while positive, suggests room for improvement in predictive accuracy.
  • Future Work: The authors suggest several directions:
  • Developing standardized evaluation metrics specifically for LLM-based protein engineering frameworks to facilitate systematic comparisons.
  • Expanding the TourSynbio-Agent's knowledge base and integrated datasets to cover a broader range of protein engineering applications.
  • Investigating the framework's potential for autonomous experimental design and optimization to further accelerate the protein engineering cycle.

Summary updated

BibTeX
@inproceedings{shen2024validation,
  author = {Yiqing Shen and Zan Chen and Yungeng Liu and Yu Guang Wang and Yiqing Shen},
  title = {{Validation of an LLM-based Multi-Agent Framework for Protein Engineering in Dry Lab and Wet Lab}},
  booktitle = {BIBM},
  year = {2024},
  doi = {10.1109/bibm62325.2024.10822562},
}
Figure