The structure-fitness landscape of pairwise relations in generative sequence models

D. Bradford Marshall, Haobo Wang, Michael A. Stiffler, Justas Dauparas, Peter K. Koo, Sergey OvchinnikovbioRxiv 2020 · DOI

TL;DR

This paper investigates how well different generative sequence models can learn both protein structure and fitness from homologous sequences. They introduce "pairwise saliency" to measure structural information and find that models excelling at one task don't necessarily excel at the other, challenging previous assumptions.

What Problem Does This Paper Address?

Understanding the relationship between a protein's sequence, its 3D structure, and its functional fitness is a fundamental challenge in computational biology. Generative sequence models aim to learn these relationships from large sets of homologous sequences. However, it's unclear how well these models, especially increasingly complex ones (from simple position-specific matrices to deep variational autoencoders), disentangle and learn both structural and fitness-related patterns. This paper tackles the question of whether models that capture structural information also capture fitness information, and vice versa, and how to quantify this.

Methods

The authors evaluate a spectrum of generative sequence models, including:

  • Position-Specific Scoring Matrices (PSSMs): These models capture only "sitewise" (per-position) conservation.
  • Markov Random Fields (MRFs): These models explicitly incorporate "pairwise" relationships, capturing coevolutionary patterns.
  • Linear Autoencoders (LAEs): These are more flexible models that can capture varying relationships, including pairwise, but don't explicitly parameterize them.
  • Variational Autoencoders (VAEs): Similar to LAEs but with non-linear activations and probabilistic latent sampling, often claimed to capture higher-order interactions.

A key novel contribution is pairwise saliency, a method to quantify the learned pairwise decomposable relations. It involves analyzing the Jacobian of the model's output with respect to its input, essentially measuring how changes in pairwise sequence relationships affect the model's predictions. This is then processed using a variant of Average Product Correction (APC) to highlight relevant contacts.

Structure prediction is evaluated using Contact AUC, which measures how well predicted residue-residue contacts match experimentally determined ones. Fitness prediction is assessed by using the models to predict the effect of mutations and then correlating these predictions with experimentally measured fitness values (using Spearman rank correlation).

Datasets

The study uses data from the TEM-1 Beta-Lactamase protein, a well-studied model system.

  • Sequence Data: Two multiple sequence alignments (MSAs) of TEM-1 Beta-Lactamase homologs were used. One was sourced from Riesselman et al. [2018] and the other generated using HMMER from a metagenomics database.
  • Structure Data: 42 experimentally determined X-ray crystal structures of TEM-1 Beta-Lactamase were obtained from the Beta-Lactamase DataBase (BLDB). These were processed to generate contact maps.
  • Fitness Data: Nine datasets of experimentally measured single-mutant fitness values were collected. These assays measure various aspects of the enzyme's function and stability under different conditions (e.g., ampicillin resistance at different concentrations, amoxicillin MIC, and thermostability).

Key Insights and Ideas

  • Structure and Fitness are Decoupled: The most striking finding is that models excelling at predicting protein structure do not necessarily excel at predicting mutation effects on fitness, and vice versa. This challenges the implicit assumption in some prior work that learning one automatically implies learning the other.
  • Pairwise Saliency as a Structural Probe: The novel pairwise saliency method provides a principled way to visualize and quantify the pairwise structural information learned by generative models, offering a more interpretable view than just prediction accuracy.
  • Model Complexity vs. Performance: While more complex models like VAEs can learn fitness well, they don't always outperform simpler models like MRFs in structure prediction. This suggests that the specific parameterization and learning objective are crucial.
  • Fitness Assay Variability: The study highlights significant variability and lack of consensus across different fitness assays, making it challenging to establish a single "ground truth" for fitness. This inconsistency can impact how well models learn fitness.

Key Results

  • MRFs: Performed well on both structure prediction (near experimental error) and fitness prediction.
  • VAEs: Showed strong performance in fitness prediction but variable performance in structure prediction.
  • LAEs: Spanned a range of performance on both tasks, generally not reaching the peak performance of MRFs for structure or VAEs for fitness.
  • PSSMs: As expected, did not capture pairwise relations and performed poorly on structure prediction.
  • The paper provides detailed plots (Figure 4) showing the trade-offs between structure prediction (Contact AUC) and fitness prediction (Spearman correlation with a specific ampicillin fitness assay) across a range of hyperparameters for each model type.

Limitations and Future Work

  • Pairwise Saliency Limitations: The authors acknowledge that pairwise saliency might not fully resolve learned contacts for highly complex models, suggesting potential for further refinement.
  • Fitness Assay Robustness: The inherent variability and subjectivity of fitness assays are noted as a significant challenge. The authors suggest that better metrics for fitness might be needed.
  • Intra-sequence Dependencies: The role of intra-sequence dependencies (beyond pairwise) for fitness inference in an unsupervised framework remains an open question.
  • Future Directions: The authors propose exploring hybrid models that combine coevolutionary information with hierarchical mixture models for phylogeny. They also suggest extending pairwise saliency to disentangle sitewise terms and to analyze larger language models for protein sequences.

Summary updated

Abstract

Abstract If disentangled properly, patterns distilled from evolutionarily related sequences of a given protein family can inform their traits - such as their structure and function. Recent years have seen an increase in the complexity of generative models towards capturing these patterns; from sitewise to pairwise to deep and variational. In this study we evaluate the degree of structure and fitness patterns learned by a suite of progressively complex models. We introduce pairwise saliency, a novel method for evaluating the degree of captured structural information. We also quantify the fitness information learned by these models by using them to predict the fitness of mutant sequences and then correlate these predictions against their measured fitness values. We observe that models that inform structure do not necessarily inform fitness and vice versa, contrasting recent claims in this field. Our work highlights a dearth of consistency across fitness assays as well as divergently provides a general approach for understanding the pairwise decomposable relations learned by a given generative sequence model.
BibTeX
@misc{marshall2020structure,
  author = {D. Bradford Marshall and Haobo Wang and Michael A. Stiffler and Justas Dauparas and Peter K. Koo and Sergey Ovchinnikov},
  title = {{The structure-fitness landscape of pairwise relations in generative sequence models}},
  year = {2020},
  doi = {10.1101/2020.11.29.402875},
  howpublished = {bioRxiv},
}
Figure