Overcoming Extrapolation Challenges of Deep Learning by Incorporating Physics in Protein Sequence-Function Modeling

Shrishti Barethiya, Jian Huang, Xiao Liu, Hui Guan, Jianhan ChenbioRxiv 2025 · DOI

Abstract

Understanding protein sequence-to-function relationship is crucial to assist studies of genetic diseases, protein evolution, and protein engineering. The sequence-to-function relationship of proteins is inherently complex due to multi-site high-dimensional correlation and structural dynamics. Deep learning algorithms such as (graph) convolutional neural networks and recently transformers have become very popular for learning the protein sequence-to-function mapping from deep mutational scanning data and available structures. However, it remains very challenging for these models to achieve accurate extrapolation when predicting functional effect of variants with positions or mutation types not seen in the training data. We propose that incorporating the physics of protein interactions and dynamics can be an effective approach to overcome the extrapolation limitations. Specifically, we demonstrate that physics-based modeling can be used to quantify the energetic effects of mutations and that incorporating these physical energetics directly within the convolution and graph convolution neural networks can significantly improve the performance of positional and mutational extrapolation compared to models without biophysics-inspired features. Our results support the effectiveness of leveraging physical knowledge in overcoming the limitation of data scarcity.

Introduction

In protein engineering, predicting how specific mutations alter a protein's function is a fundamental challenge. Deep Mutational Scanning (DMS) has enabled the generation of large-scale datasets, but the astronomical size of sequence space means even the most comprehensive experiments cover only a fraction of possible variants. While standard deep learning models excel at interpolation—predicting effects similar to those in the training set—they struggle with extrapolation. Specifically, they often fail when predicting the effects of mutation types not seen during training (mutational extrapolation) or mutations at entirely new positions (positional extrapolation). This "extrapolation gap" arises because these models lack an inherent understanding of the physical laws governing protein folding and stability.

Methods: Integrating Physics into Neural Networks

To address these limitations, we developed a hybrid approach that augments standard neural network architectures with biophysical features.

Physics-Based Features

  1. Rosetta Energetics: We utilized the Rosetta software suite to calculate the change in folding stability ($\Delta\Delta G$) for all possible single-point mutations. Rather than using a single aggregate score, we decomposed the energy into eight distinct components, including hydrogen bonding, electrostatic potential, and steric clashes.
  2. Molecular Dynamics (MD): To account for protein flexibility, we performed 200-nanosecond simulations to calculate the Root Mean Square Fluctuation (RMSF), identifying highly dynamic regions of the protein structure.

Model Architectures

These features were integrated into two primary architectures: Convolutional Neural Networks (CNNs), which treat the protein sequence as a 1D input, and Graph Convolutional Networks (GCNs), which represent the protein as a 3D graph where residues are nodes connected based on spatial proximity in the folded structure.

Datasets and Benchmark Proteins

The proposed models were evaluated across five diverse proteins, representing a range of sizes, functions, and data coverage levels.

Structural diversity and data coverage of the five benchmark proteins (GB1, Pab1, Ube4b, avGFP, and Bgl3).
Summary of protein characteristics and dataset sizes used for benchmarking extrapolation.

The proteins range from the small, well-covered GB1 domain to the larger, more complex Bgl3 enzyme. This diversity allows for a robust assessment of how physics-informed features handle varying degrees of data scarcity and structural complexity.

Key Results and Insights

The incorporation of physical priors significantly enhanced the models' ability to generalize beyond the training data.

Overcoming the Extrapolation Barrier

  • Positional Extrapolation: In tasks where models had to predict mutations at completely unseen sites, standard architectures often showed near-zero correlation. Adding physics features improved the CNN's Pearson correlation on GB1 from 0.05 to 0.43.
  • Mutational Extrapolation: For avGFP, the addition of physics features boosted correlation from 0.06 to 0.61, demonstrating that physical rules (like steric hindrance) provide a crucial "grounding" for the model.

Efficiency and Data Scarcity

Physics-informed models proved particularly resilient in data-scarce scenarios. By providing the model with "cheat sheets" of energetic costs, it can learn meaningful patterns from fewer experimental examples. Furthermore, this approach achieved performance comparable to massive Transformer-based models (e.g., METL) pre-trained on millions of sequences, suggesting that explicit physical knowledge can be as effective as massive-scale data mining.

Conclusion and Future Work

While incorporating physics significantly improves extrapolation, challenges remain. For instance, the dynamic nature of proteins like Ube4b still poses difficulties for current physics-based tools. Future research will focus on moving beyond the assumption of mutational additivity to better capture epistasis—the complex interactions between multiple mutations. Nonetheless, this work demonstrates that bridging the gap between deep learning and biophysics is a powerful strategy for navigating the vast landscape of protein sequence space.

Summary updated

Abstract

Understanding protein sequence-to-function relationship is crucial to assist studies of genetic diseases, protein evolution, and protein engineering. The sequence-to-function relationship of proteins is inherently complex due to multi-site high-dimensional correlation and structural dynamics. Deep learning algorithms such as (graph) convolutional neural networks and recently transformers have become very popular for learning the protein sequence-to-function mapping from deep mutational scanning data and available structures. However, it remains very challenging for these models to achieve accurate extrapolation when predicting functional effect of variants with positions or mutation types not seen in the training data. We propose that incorporating the physics of protein interactions and dynamics can be an effective approach to overcome the extrapolation limitations. Specifically, we demonstrate that physics-based modeling can be used to quantify the energetic effects of mutations and that incorporating these physical energetics directly within the convolution and graph convolution neural networks can significantly improve the performance of positional and mutational extrapolation compared to models without biophysics-inspired features. Our results support the effectiveness of leveraging physical knowledge in overcoming the limitation of data scarcity.
BibTeX
@misc{barethiya2025overcoming,
  author = {Shrishti Barethiya and Jian Huang and Xiao Liu and Hui Guan and Jianhan Chen},
  title = {{Overcoming Extrapolation Challenges of Deep Learning by Incorporating Physics in Protein Sequence-Function Modeling}},
  year = {2025},
  doi = {10.1101/2025.11.09.687530},
  howpublished = {bioRxiv},
}
Figure