BERT and LLMs-Based avGFP Brightness Prediction and Mutation Design

Xuejiang Guo, Wanxiang ChearXiv 2024 · DOI · arXiv

TL;DR

Researchers have developed a hybrid machine learning pipeline that combines specialized Transformer models with the generative power of LLMs (like GPT and Claude) to predict and design high-brightness variants of Green Fluorescent Protein (avGFP). By using a "scoring" model to evaluate sequences and LLMs to suggest biologically plausible mutations, the team identified 10 promising new mutants that could significantly improve bio-imaging efficiency.

What Problem Does This Paper Address?

The Green Fluorescent Protein (GFP) is the "microscope of the 21st century," allowing scientists to watch cellular processes in real-time. However, the wild-type protein isn't always bright enough or stable enough for every application.

Traditionally, creating better versions of GFP requires "directed evolution"—essentially a high-speed version of natural selection in a test tube. This process is incredibly slow, expensive, and limited by how many millions of physical samples a lab can screen. The authors of this paper aim to move this process into the digital realm, using deep learning to navigate the astronomical number of possible protein sequences and identify "super-bright" mutants before a single drop of liquid is touched in a lab.

Methods

The study employs a two-stage "Design-and-Verify" methodology:

  1. The Scorer (Transformer Model): The authors built a Transformer-based architecture (similar to BERT) specifically for protein sequences.
  2. Preprocessing: Sequences were padded to a uniform length of 239 amino acids and mapped to numerical values.
  3. The "Secret Sauce": They implemented a mutation mask to highlight specific change sites, helping the model focus on how substitutions affect the overall protein structure.
  4. Architecture: The model uses embedding layers, multiple Transformer layers for encoding, and average pooling to output a predicted brightness value. Interestingly, they found that "position encoding"—a standard feature in most Transformers—didn't improve performance here and was omitted.
  5. The Designer (LLM Integration): Instead of just randomly mutating sequences, the authors used GPT and Claude as "creative consultants."
    • They first performed statistical analysis on their dataset to find "hotspot" mutation sites.
  6. These hotspots were fed to LLMs to generate ~200 novel mutation states based on the models' internal "knowledge" of protein chemistry.
  7. The Transformer Scorer then ranked these 200 candidates.
  8. Finally, the LLMs were used again to cross-reference the top candidates against existing scientific literature to ensure the mutations made biological sense (e.g., checking for $\pi-\pi$ interactions or $\beta$-barrel stability).

Datasets

The study utilized a proprietary dataset containing approximately 140,000 protein sequences. Within this set, roughly 30,000 were specific avGFP sequences with associated brightness labels. This large-scale dataset allowed the Transformer model to learn the complex relationship between amino acid sequences and optical properties.

Key Insights and Ideas

  • The Hybrid Advantage: The most striking idea is the "loop" between specialized ML and general LLMs. While the Transformer model is great at precise scoring, LLMs are better at "reasoning" about biological context and suggesting creative mutations that might not be obvious from raw statistics alone.
  • Structural Awareness: The model successfully identified that mutations near the chromophore (the light-emitting part of the protein) have the highest impact. For example, the top-ranked mutant (V162C) was chosen because the introduced cysteine could form new interactions that alter electronic transition properties.
  • Beyond the Sequence: By using LLMs for the final selection, the researchers integrated "prior knowledge" about protein folding and stability—factors that a simple regression model trained only on brightness might ignore.

Key Results

The study culminated in the design of 10 high-brightness avGFP sequences.

  • Top Performer: The V162C single-point mutation achieved the highest predicted brightness score of 3.816.
  • Complex Mutants: The model also designed sophisticated multi-point mutants, such as a five-point mutant (S202Y/K157G/I170V/N104S/V162A) with a predicted brightness of 3.779.
  • Biological Logic: The reasons provided for these selections—such as reducing steric hindrance (V162A) or introducing positive charges to stabilize the chromophore (Q183R)—align well with established principles of protein engineering.

Limitations and Future Work

The primary limitation of this paper is the lack of "wet-lab" validation. While the models predict high brightness, these 10 sequences have not yet been synthesized and tested in living cells.

Future directions suggested by the authors include: * Experimental Validation: Using CRISPR and other genome-editing tools to actually produce these proteins and measure their real-world brightness. * Multimodal Approaches: Integrating other factors like folding efficiency, thermal stability, and photostability into the prediction model. * Broader Data: Incorporating more diverse experimental datasets to improve the model's ability to generalize across different types of fluorescent proteins.

Summary updated

Abstract

This study aims to utilize Transformer models and large language models (such as GPT and Claude) to predict the brightness of Aequorea victoria green fluorescent protein (avGFP) and design mutants with higher brightness. Considering the time and cost associated with traditional experimental screening methods, this study employs machine learning techniques to enhance research efficiency. We first read and preprocess a proprietary dataset containing approximately 140,000 protein sequences, including about 30,000 avGFP sequences. Subsequently, we constructed and trained a Transformer-based prediction model to screen and design new avGFP mutants that are expected to exhibit higher brightness. Our methodology consists of two primary stages: first, the construction of a scoring model using BERT, and second, the screening and generation of mutants using mutation site statistics and large language models. Through the analysis of predictive results, we designed and screened 10 new high-brightness avGFP sequences. This study not only demonstrates the potential of deep learning in protein design but also provides new perspectives and methodologies for future research by integrating prior knowledge from large language models.
BibTeX
@misc{guo2024bert,
  author = {Xuejiang Guo and Wanxiang Che},
  title = {{BERT and LLMs-Based avGFP Brightness Prediction and Mutation Design}},
  year = {2024},
  eprint = {2407.20534},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.20534},
}
Figure