Efficient Generative Modelling of Protein Structure Fragments using a Deep Markov Model

Christian B. Thygesen, Christian Skjødt Steenmans, Ahmad Salim Al-Sibahi, Lys Sanz Moreta, Anders Bundgård Sørensen, Thomas HamelryckICML 2021 · OpenReview

Predicting how a protein folds is a monumental computational challenge because the "conformational search space"—the number of possible shapes a protein chain can take—is astronomically large. To make this manageable, researchers often use a "divide-and-conquer" strategy: they break the protein into short segments (fragments) and sample likely local shapes for these segments from a "fragment library."

Traditional tools, like the Rosetta fragment picker, generate these libraries by searching databases for similar sequences and using external secondary structure predictions. However, these methods are computationally expensive, struggle when homologous sequences aren't available (e.g., for synthetic or "orphan" proteins), and don't explicitly account for uncertainty—both the experimental errors in data (epistemic) and the inherent flexibility of proteins (aleatory).

To address these limitations, researchers have developed BIFROST (Bayesian Inference for FRagments Of protein STructures), a novel Deep Markov Model that generates protein backbone fragment libraries using only amino acid sequences as input. By combining deep learning with directional statistics, it achieves state-of-the-art accuracy at a fraction of the runtime of traditional methods while providing a principled way to quantify structural uncertainty.

A Deep Probabilistic Approach

BIFROST is a generative model based on a Deep Markov Model (DMM) architecture, implemented in the deep probabilistic programming language Pyro. This allows the use of Stochastic Variational Inference (SVI) to estimate model parameters.

The BIFROST generative model architecture showing the flow from amino acid sequence (a) through hidden states (h) and latent variables (z) to observed angles (phi, psi).

The model introduces several key technical innovations:

  • Sequence-Only Conditioning: Unlike traditional methods that require multiple sequence alignments or structural templates, BIFROST generates backbone dihedral angles ($\phi$ and $\psi$) conditioned solely on the primary amino acid sequence.
  • Directional Statistics: Because angles are periodic (e.g., 359° is close to 1°), standard Gaussian distributions are inadequate. BIFROST uses a bivariate periodic Student-T distribution. This "wrapped" distribution is more robust to outliers than the traditional von Mises distribution and performs better during training.
  • Bidirectional Architecture: The model uses a bidirectional Recurrent Neural Network (RNN) to process the amino acid sequence, capturing information from both "upstream" and "downstream" residues. This information feeds into transition and emitter neural networks that generate the latent states and the resulting backbone angles.

Training and Datasets

BIFROST was trained on a high-quality subset of the cullpdb dataset, containing 3,733 proteins. The authors applied strict quality filters, including a resolution < 1.6Å and an R-factor < 0.25. To ensure the model learned generalizable features rather than just "memorizing" similar proteins, a sequence identity cutoff of 20% was applied. Furthermore, sequences with > 20% identity to CASP13 targets were removed to ensure fair downstream evaluation. The final training set consisted of approximately 186,000 9-mer fragments.

Key Results and Insights

By framing fragment generation as a probabilistic task, BIFROST offers several distinct advantages over traditional database-search methods:

  • Uncertainty as a Feature: Because BIFROST is probabilistic, it doesn't just output a single "best guess" for a structure; it provides a probability distribution. This allows researchers to evaluate the likelihood of a specific local structure, which is vital for accurate physics-based simulations.
  • Capturing Chemical Nuance: The model successfully learns the unique structural constraints of specific amino acids without explicit programming. For example, it correctly models the high flexibility of Glycine and the rigid, restricted conformational space of Proline.
  • Performance Parity and Speedup: BIFROST generates fragment libraries with precision and coverage comparable to the industry-standard Rosetta fragment picker, despite using significantly less input information. Furthermore, by amortizing the "search" into the neural network's weights, generating a library becomes a simple forward pass. While both BIFROST and Rosetta scale linearly with protein length, BIFROST has a much smaller constant term, making it significantly faster for large-scale structural studies.
  • Structural Fidelity: The model accurately recreates the "Ramachandran plots"—the distribution of allowed backbone angles—observed in real experimental data, demonstrating its ability to capture realistic protein backbone behavior.
Modelled Ramachandran plot showing BIFROST's ability to capture the general distribution of protein backbone dihedral angles.

Limitations and Future Directions

While highly effective, the current iteration of BIFROST has a few limitations. First, although the DMM architecture allows for arbitrary fragment lengths, performance tends to drop off as fragments get longer. Second, the model currently assumes the $\omega$ dihedral angle is fixed at 180°. While this is usually true, it means rare cis-peptide bonds are not captured. Finally, BIFROST's predicted distributions are slightly wider than those derived directly from the Protein Data Bank (PDB). The authors suggest this may reflect real protein dynamics not captured in static X-ray structures, though it could also indicate slight model under-fitting.

Future work could involve integrating BIFROST directly into folding protocols like Rosetta to determine if its probabilistic weights and rapid generation times can improve the accuracy and efficiency of full-length protein structure predictions.

Summary updated

Abstract

Fragment libraries are often used in protein structure prediction, simulation and design as a means to significantly reduce the vast conformational search space. Current state-of-the-art methods for fragment library generation do not properly account for aleatory and epistemic uncertainty, respectively due to the dynamic nature of proteins and experimental errors in protein structures. Additionally, they typically rely on information that is not generally or readily available, such as homologous sequences, related protein structures and other complementary information. To address these issues, we developed BIFROST, a novel take on the fragment library problem based on a Deep Markov Model architecture combined with directional statistics for angular degrees of freedom, implemented in the deep probabilistic programming language Pyro. BIFROST is a probabilistic, generative model of the protein backbone dihedral angles conditioned solely on the amino acid sequence. BIFROST generates fragment libraries with a quality on par with current state-of-the-art methods at a fraction of the run-time, while requiring considerably less information and allowing efficient evaluation of probabilities.
BibTeX
@inproceedings{thygesen2021efficient,
  author = {Christian B. Thygesen and Christian Skjødt Steenmans and Ahmad Salim Al-Sibahi and Lys Sanz Moreta and Anders Bundgård Sørensen and Thomas Hamelryck},
  title = {{Efficient Generative Modelling of Protein Structure Fragments using a Deep Markov Model}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=tRGAw3xhVr},
}
Figure