Extrapolating Protein Fitness Beyond the Training Distribution

64 concepts50 papers

When applying machine learning to protein engineering, we face a fundamental paradox. Standard machine learning models thrive on interpolation—predicting properties of unseen data points that lie comfortably within the boundaries of the training data. In protein design, however, our goal is almost always extrapolation. We do not want to discover a protein that is merely as good as the ones we have already sequenced; we want to discover variants that exhibit superior stability, tighter binding affinity, or novel catalytic functions that outperform everything in our training set.

Navigating this out-of-distribution (OOD) space is exceptionally difficult. The combinatorial space of amino acid sequences is astronomically large, and the vast majority of it is non-functional. To successfully engineer new proteins, models must break through the "fitness ceiling" of their training data and generalize across massive mutational gaps.

The Illusion of In-Distribution Success

Why can't we just train a standard, highly accurate regression model on existing mutational data and use it to find better proteins? The answer lies in how we evaluate these models.

In a typical machine learning workflow, data is split randomly into training and validation sets. As demonstrated by Michaël Richard et al. in A systematic analysis of regression models for protein engineering, random cross-validation severely overestimates a model's real-world utility. When a model is evaluated using random splits, it excels because it interpolates between closely related sequence variants. However, when tasked with predicting fitness at entirely unseen sequence positions or evaluating higher-order mutations, the model's performance frequently collapses. Rather than learning the underlying biophysical principles of the protein, the model simply memorizes site-specific signals from the training set.

This generalization gap manifests along several distinct axes:

  • The Fitness Ceiling: When training data is restricted to low- or medium-fitness sequences, models struggle to predict variants that surpass that range. As curated by Jody Mou et al. in the benchmarking framework FLIP: Benchmark tasks in fitness landscape inference for proteins, models that score perfectly on interpolative tasks often fail entirely when required to extrapolate to higher activity levels than those seen during training.
  • The Mutational Gap and Epistasis: High-throughput experimental assays typically generate sequences that are only one or two mutations away from a known "wild-type" starting point. However, combining multiple beneficial mutations introduces epistasis—non-linear interactions where the combined effect of mutations differs from the sum of their individual parts. Nathaniel Charest et al. show in Discovering pathways through ribozyme fitness landscapes using information theoretic quantification of epistasis that the predictive power of local mutational data degrades rapidly as the distance from the training set increases.
  • Structural and Functional Novelty: When models are forced to design sequences for synthetic or de novo protein structures that do not exist in nature, they hit a wall. Zhangyang Gao et al. document this performance gap in ProteinInvBench: Benchmarking Protein Inverse Folding on Diverse Tasks, Models, and Metrics, showing that sequence-structure models fail to generalize when stripped of natural evolutionary templates.

Smoothing the Rugged Landscape

To understand why models fail to extrapolate, it helps to visualize the protein fitness landscape. In nature, these landscapes are incredibly "rugged"—a single amino acid substitution can cause a protein to misfold entirely, turning a high-fitness peak into a non-functional valley. Standard models struggle to find pathways through this jagged terrain without falling into local minima or making catastrophic prediction errors in uncharted territory.

One powerful mechanism for overcoming this is landscape smoothing. If we can mathematically smooth out the high-frequency noise of the fitness landscape, optimization algorithms can detect broader evolutionary trends and extrapolate across wider mutational gaps.

Rugged Landscape (Hard to Extrapolate)      Smoothed Landscape (Clear Gradient)
         _/\_             _                 _          __--""--__
       _/    \_         _/ \_              /  \__--""--          \
  ____/        \_______/     \____     ___/                       \___

Andrew Kirjner et al. implement this intuition in Improving protein optimization with smoothed fitness landscapes using graph signal processing. By modeling sequence space as a graph and applying a low-pass filter, they smooth out the rugged cliffs. This regularization allows optimization algorithms to look past local epistatic noise and successfully discover novel sequences containing multiple simultaneous mutations.

Taking a structural view of this graph representation, Hao Zhu et al. propose a spectral graph theory framework in Protein Fitness Landscape: Spectral Graph Theory Perspective. By treating sequence space as a generalized Hamming graph, their Propagational Convolutional Neural Networks (PCNNs) smooth out noisy experimental data, enabling the model to maintain predictive accuracy even when evaluating variants far from the training distribution.

Iterative Refinement and Latent Spaces

Instead of trying to predict a highly optimized, distant sequence in a single massive leap, another core strategy breaks the extrapolation problem into a sequence of manageable, local steps.

In Extrapolative Controlled Sequence Generation via Iterative Refinement, Vishakh Padmakumar et al. demonstrate that while models fail at global extrapolation, they can achieve it incrementally. By training a model to make small, localized improvements and applying those updates iteratively, the sequence gradually evolves into high-performance territory completely unobserved in the initial training data.

To ensure these iterative jumps do not land in non-functional regions of sequence space, researchers often map discrete amino acids into continuous latent spaces. Lea Bogensperger et al. leverage this approach in A Variational Perspective on Generative Protein Fitness Optimization. By combining flow-matching generative priors with manifold-constrained fitness guidance, their framework effectively "hallucinates" high-fitness variants. The continuous latent space acts as a safety rail, keeping the extrapolative search bounded within viable biological configurations.

Contextualizing Sparse Signals

When data is scarce, a model cannot afford to look at a target sequence in isolation. It must leverage contextual clues from other known variants to guide its out-of-distribution predictions.

To maximize the utility of limited data, Yarin Gal introduces a non-parametric transformer architecture in ProteinNPT: Improving Protein Property Prediction and Design with Non-Parametric Transformers. Unlike standard architectures that process sequences independently, this model attends simultaneously across the residues within a sequence and across all other labeled examples within a training batch. This allows the model to dynamically borrow strength from related variants, making it far more robust when predicting mutations at sequence positions or combinations that were entirely absent from the training set.

Standard Transformer:     [ Sequence X ] ----------> Predict Fitness
                               ^
Non-Parametric (NPT):     [ Sequence X ] <---------> [ Labeled Variant A ]
                               ^ <---------> [ Labeled Variant B ]

An alternative way to handle this distribution shift is to adapt the model dynamically at the moment of prediction. Anton Bushuiev et al. introduce this paradigm in Training on test proteins improves fitness, structure, and function prediction. Instead of relying on a static pre-trained model, their Test-Time Training (TTT) approach optimizes the model's internal parameters on the unlabelled target sequence before making a fitness prediction. By minimizing its internal "surprise" or perplexity on the target, the model shifts its own representation space closer to the out-of-distribution protein, mitigating the performance collapse that typically plagues unfamiliar protein families.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_out_of_distribution_fitness_prediction_7ab01930,
  title = {out-of-distribution fitness prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/out-of-distribution-fitness-prediction/rdr_7ab01930fb4f42ed9889f4c8f60310d7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_out_of_distribution_fitness_prediction_7ab01930_v1,
  title = {out-of-distribution fitness prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/out-of-distribution-fitness-prediction/rdr_7ab01930fb4f42ed9889f4c8f60310d7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake