ProteinEngine: Empower LLM with Domain Knowledge for Protein Engineering

Yiqing Shen, Outongyi Lv, Houying Zhu, Yu Guang Wang

Abstract

Protein engineering is a high-stakes field requiring deep expertise in structure prediction, evolution analysis, and molecular docking. While Large Language Models (LLMs) show promise as general assistants, they lack the domain-specific technical knowledge and the ability to interface with specialized bioinformatics tools. We present ProteinEngine, a multi-agent framework that transforms LLMs into specialized assistants for protein engineering. By delegating tasks to a Project Manager, Domain Expert, and Presenter, the system orchestrates complex biological tools (e.g., AlphaFold, DiffDock) through natural language. Our user study involving 45 participants demonstrates that ProteinEngine significantly reduces time and technical barriers, improving accuracy and usability in protein design and analysis.

Introduction

The process of developing valuable proteins—protein engineering—is traditionally bottlenecked by two major challenges. First, knowledge scarcity: general-purpose LLMs like GPT-4 lack the hyper-specific, up-to-date technical data required for actual biological computations. Second, tool fragmentation: specialized tools such as AlphaFold for folding or PyMOL for visualization require complex setups, specific input formats, and programming proficiency. These factors create a steep "syntax tax" for researchers.

Previous attempts to specialize LLMs via fine-tuning often degrade the model's natural conversational abilities or require prohibitively expensive datasets. ProteinEngine addresses these issues by giving LLMs "hands" to use existing specialized tools without losing their "brain" for communication, utilizing a human-centered, multi-agent orchestration approach.

The ProteinEngine Framework

ProteinEngine employs a role-playing framework to manage complex workflows. Rather than relying on a single LLM to perform all tasks, the system splits responsibilities among three distinct agents:

  • AI Project Manager (AI-PM): Acting as the primary interface, the AI-PM interprets natural language requests, identifies necessary sub-tasks, and ensures all required parameters (such as PDB files or sequences) are present before execution.
  • AI Domain Expert (AI-DE): This specialist module manages a library of APIs for state-of-the-art (SOTA) models, including ESMFold, GearNet, and DiffDock. It features a self-feedback loop with the PM to refine task understanding or request missing data if an execution fails.
  • AI Presenter (AI-Pr): The AI-Pr aggregates raw data from the experts and converts it into user-friendly formats. This includes 3D visualizations via PyMOL or VMD and concise summaries, acknowledging that researchers require spatial data to make informed decisions.

The system utilizes in-context learning (prompting) rather than fine-tuning. This makes ProteinEngine highly extensible; new tools can be integrated simply by defining a new API call for the Domain Expert.

Experimental Methodology

To evaluate the efficacy of ProteinEngine, the authors conducted a user study with 45 participants from both AI and protein engineering backgrounds across academia and industry. Participants were assigned six distinct tasks, including protein folding, inverse folding, and mutation prediction.

The study compared ProteinEngine against a manual baseline where users were required to execute Python scripts and manage tools independently. Performance was measured across four hypotheses: time efficiency, accuracy, usability, and perceived workload.

Results and Discussion

The researchers used single-sided t-tests to compare the framework against the baseline. The results were statistically significant ($p < 0.05$) for three of the four metrics:

  1. Time Efficiency: ProteinEngine significantly reduced the time required to identify and execute models ($p = 2.335 \times 10^{-8}$).
  2. Accuracy: Users made significantly fewer errors in identifying and executing the correct models ($p = 1.884 \times 10^{-3}$).
  3. Usability: The system received significantly higher System Usability Scale (SUS) scores ($p = 1.166 \times 10^{-2}$).
  4. Workload: Interestingly, there was no significant difference in the perceived workload (NASA TLX index). While the system was faster and more accurate, users still felt the cognitive load of the complex biological tasks themselves, suggesting that while the "syntax tax" was removed, the scientific complexity remained.

Conclusion and Future Work

ProteinEngine demonstrates that the most effective way to utilize LLMs in specialized scientific fields is as an "orchestrator" rather than a standalone solver. By acting as a middleman between scientists and complex scripts, the framework removes technical barriers to entry.

However, limitations remain. The system's reasoning is currently capped by the underlying LLM (primarily gpt-3.5-turbo in this study). Furthermore, the ease of use introduced by ProteinEngine necessitates the development of ethical guardrails to prevent the misuse of technology for harmful protein design. Future work will focus on integrating more robust validation protocols to ensure that AI-generated designs are biologically viable and lab-ready.

Summary updated

BibTeX
@inproceedings{shen2024proteinengine,
  author = {Yiqing Shen and Outongyi Lv and Houying Zhu and Yu Guang Wang},
  title = {{ProteinEngine: Empower LLM with Domain Knowledge for Protein Engineering}},
  booktitle = {AIME},
  year = {2024},
  doi = {10.1007/978-3-031-66538-7\_37},
}
Figure