Guided Generative Protein Design using Regularized Transformers

Egbert Castro, Abhinav Godavarthi, Julian Rubinfien, Kevin B. Givechian, Dhananjay Bhaskar, Smita Krishnaswamy

Abstract

The development of powerful natural language models has increased the ability to learn meaningful representations of protein sequences. In addition, advances in high-throughput mutagenesis, directed evolution, and next-generation sequencing have allowed for the accumulation of large amounts of labeled fitness data. Leveraging these two trends, we introduce Regularized Latent Space Optimization (ReLSO), a deep transformer-based autoencoder which features a highly structured latent space that is trained to jointly generate sequences as well as predict fitness. Through regularized prediction heads, ReLSO introduces a powerful protein sequence encoder and novel approach for efficient fitness landscape traversal. Using ReLSO, we explicitly model the sequence-function landscape of large labeled datasets and generate new molecules by optimizing within the latent space using gradient-based methods. We evaluate this approach on several publicly-available protein datasets, including variant sets of anti-ranibizumab and GFP. We observe a greater sequence optimization efficiency (increase in fitness per optimization step) by ReLSO compared to other approaches, where ReLSO more robustly generates high-fitness sequences. Furthermore, the attention-based relationships learned by the jointly-trained ReLSO models provides a potential avenue towards sequence-level fitness attribution information.

Introduction

Protein design is a "needle in a haystack" problem of astronomical proportions. Even a small protein of 30 residues presents a search space of $10^{38}$ possible sequences, far exceeding the capacity of modern laboratory screening. This challenge is compounded by epistasis—complex, non-linear interactions where mutations at one site affect the function of residues elsewhere in the protein.

Traditional methods like directed evolution are often slow, expensive, and prone to becoming trapped in local optima. While machine learning has improved fitness prediction, navigating the discrete, high-dimensional space of amino acids remains a significant bottleneck. ReLSO addresses this by transforming discrete sequence search into a continuous, navigable "fitness landscape."

Methods

The ReLSO framework (Regularized Latent Space Optimization) utilizes a transformer-based autoencoder to map protein sequences into a smooth, highly structured latent space.

ReLSO architecture: A transformer encoder maps protein sequences to a latent space embedding (z), which is used to model the fitness landscape.

Key components of the architecture include:

  1. Transformer-based Encoder: Uses attention mechanisms to capture long-range interactions (epistasis) essential for protein folding and function.
  2. Information Bottleneck: Forces protein data through a low-dimensional latent space ($z$), ensuring only the most salient features are preserved.
  3. Joint Training (JT-AE): The model is trained to simultaneously reconstruct sequences and predict fitness (e.g., binding affinity or fluorescence), ensuring the latent space is organized by both structure and function.
  4. Negative Sampling for Pseudo-Convexity: To prevent optimization from wandering into biologically unrealistic regions, high-norm regions outside the training data are sampled and assigned low fitness. This creates a "peak" of high fitness centered on known data, making the landscape "pseudo-convex."
  5. Interpolation Regularization: This penalty ensures that small steps in the latent space correspond to gradual, meaningful changes in the protein sequence, preventing disjointed or nonsensical outputs.

Results and Key Insights

ReLSO was evaluated on several key protein benchmarks, including the Gifford (anti-ranibizumab antibodies), GB1 (immunoglobulin-binding protein), and GFP (Green Fluorescent Protein) datasets, as well as the TAPE benchmark for representation learning.

Optimization Efficiency and Navigability

A primary advantage of ReLSO is its ability to create a "smooth" landscape that is highly amenable to gradient-based optimization. Unlike standard Jointly-Trained Autoencoders (JT-AE), ReLSO's regularizations produce a landscape where gradient ascent consistently leads toward high-fitness regions.

Comparison of model fitness landscapes and gradient ascent trajectories between JT-AE and ReLSO. ReLSO's structured landscape allows for more directed and efficient optimization.

Key findings include:

  • The "Trust Boundary": Negative sampling creates an implicit "trust boundary," keeping the optimizer focused on regions near the training data and mitigating the risk of "model exploitation" (finding sequences the model thinks are good but are actually biologically impossible).
  • Robustness: In the GFP dataset, which features a challenging bimodal fitness distribution, ReLSO successfully navigated the landscape to consistently identify high-fluorescence modes.
  • Generative Power: For the anti-ranibizumab antibody task, ReLSO generated over 11,000 novel variants. When mapped back into the latent space, these localized to high-fitness regions, suggesting the model successfully learned the underlying rules of antibody binding.

Discussion

While ReLSO shows great promise, the authors note that for datasets with very low sequence diversity (like GB1), it is difficult to meaningfully assess the model's reconstruction ability. However, the framework's success in navigating complex landscapes suggests it is highly translatable to other biomolecules like DNA and RNA. Future applications may include "tuning" protein-protein binding to increase selectivity, which could help mitigate off-target toxicity in drug development—a critical goal in biotherapeutics.

Summary updated

Abstract

The development of powerful natural language models have increased the ability to learn meaningful representations of protein sequences. In addition, advances in high-throughput mutagenesis, directed evolution, and next-generation sequencing have allowed for the accumulation of large amounts of labeled fitness data. Leveraging these two trends, we introduce Regularized Latent Space Optimization (ReLSO), a deep transformer-based autoencoder which features a highly structured latent space that is trained to jointly generate sequences as well as predict fitness. Through regularized prediction heads, ReLSO introduces a powerful protein sequence encoder and novel approach for efficient fitness landscape traversal. Using ReLSO, we explicitly model the sequence-function landscape of large labeled datasets and generate new molecules by optimizing within the latent space using gradient-based methods. We evaluate this approach on several publicly-available protein datasets, including variant sets of anti-ranibizumab and GFP. We observe a greater sequence optimization efficiency (increase in fitness per optimization step) by ReLSO compared to other approaches, where ReLSO more robustly generates high-fitness sequences. Furthermore, the attention-based relationships learned by the jointly-trained ReLSO models provides a potential avenue towards sequence-level fitness attribution information.
BibTeX
@misc{castro2022guided,
  author = {Egbert Castro and Abhinav Godavarthi and Julian Rubinfien and Kevin B. Givechian and Dhananjay Bhaskar and Smita Krishnaswamy},
  title = {{Guided Generative Protein Design using Regularized Transformers}},
  year = {2022},
  eprint = {2201.09948},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.09948},
}
Figure