TL;DR
This paper introduces ReLSO, a novel transformer-based model that learns a structured latent space for protein sequences, enabling efficient optimization and generation of high-fitness proteins. By jointly training sequence generation and fitness prediction, ReLSO navigates complex protein fitness landscapes more effectively than existing methods, paving the way for improved protein design in various applications.
What Problem Does This Paper Address?
Designing novel proteins with specific functions is a grand challenge due to the astronomically vast search space of possible amino acid sequences. Traditional methods like directed evolution are often slow and can get stuck in local optima. While deep learning has shown promise in predicting protein fitness, efficiently navigating these complex "fitness landscapes" to design new, high-performing proteins remains difficult. Existing approaches often struggle with the inherent ruggedness of these landscapes, where small sequence changes can lead to unpredictable fitness shifts, and can be inefficient in exploring the space to find truly optimal sequences.
Methods
ReLSO builds upon the power of transformer architectures, commonly used in natural language processing, to encode protein sequences into a low-dimensional "latent space." The key innovation lies in how this latent space is structured and utilized:
- Joint Training: ReLSO is trained to simultaneously reconstruct protein sequences and predict their fitness. This "jointly-trained autoencoder" (JT-AE) approach forces the latent space to capture information about both sequence identity and function.
- Regularized Latent Space: To make the latent space amenable to optimization, ReLSO employs several novel regularizations:
- Negative Sampling: This technique artificially introduces low-fitness points around real data points, reshaping the fitness landscape to be more "pseudo-convex" and providing a natural stopping criterion for optimization.
- Interpolative Sampling: This ensures that small steps in the latent space correspond to gradual changes in the decoded protein sequence, allowing for smoother exploration and denser sampling of the sequence manifold.
- Latent Space Optimization: Once trained, ReLSO uses gradient ascent directly in this structured latent space to find regions of high fitness. This optimized latent representation is then decoded back into a high-fitness protein sequence.
Datasets
The authors evaluated ReLSO on several publicly available protein datasets:
- Gifford Dataset: Data from directed evolution experiments on an antibody library, measuring sequence enrichment.
- GB1 Dataset: Data from a saturation mutagenesis study on an immunoglobulin-binding protein, measuring stability and binding affinity.
- GFP Dataset: Data from random mutagenesis of the Aequorea victoria Green Fluorescent Protein (GFP), measuring fluorescence.
- TAPE Dataset: A benchmark dataset for fluorescence, used for comparison.
Key Insights and Ideas
- Structured Latent Space is Key: The paper strongly suggests that simply encoding proteins isn't enough; the latent space needs to be carefully structured to facilitate efficient optimization. ReLSO's regularizations are crucial for this.
- Gradient Ascent in Latent Space Works: By creating a well-behaved latent space, gradient ascent becomes a powerful and efficient tool for protein design, outperforming other methods in optimization speed and robustness.
- Attention Maps for Interpretability: The transformer's attention mechanism in ReLSO offers a potential avenue for understanding sequence-fitness relationships, hinting at which parts of a protein are most critical for its function.
- Biochemical Properties are Recapitulated: The amino acid embeddings learned by ReLSO reflect their underlying biochemical properties, indicating that the model is learning biologically meaningful representations.
Key Results
- Higher Optimization Efficiency: ReLSO demonstrated greater "sequence optimization efficiency" (increase in fitness per optimization step) compared to other methods across multiple datasets.
- Robust Generation of High-Fitness Sequences: ReLSO was more robust in generating high-fitness sequences, particularly in more challenging landscapes like GIFFORD and GB1.
- Faster Convergence: Gradient ascent in ReLSO's latent space showed quick convergence to high-fitness regions, often achieving this in fewer optimization steps than other approaches.
- Improved Latent Space Organization: ReLSO's latent space was better organized by fitness and sequence compared to simpler autoencoder models.
Limitations and Future Work
- Computational Cost: While ReLSO is efficient in optimization steps, training the transformer-based model can be computationally intensive.
- Generalizability to Other Biomolecules: The authors suggest that ReLSO's reliance on sequence information could make it translatable to other biomolecules like DNA and RNA, potentially aiding in areas like CRISPR-Cas9 off-target effect reduction.
- Therapeutic Applications: The paper highlights the potential for ReLSO in biotherapeutic optimization, such as tuning antibody binding affinity to improve selectivity and mitigate off-target toxicity.
- Interpretability: While attention maps offer insights, further work is needed to fully elucidate sequence-fitness relationships and leverage them for design.
Summary updated