Machine learning has become remarkably adept at predicting the properties of natural proteins and DNA. However, most models suffer from a fundamental "interpolation trap": they excel at predicting sequences similar to those in their training data but fail when tasked with designing entirely new biological functions or navigating "unseen" mutational spaces.
The emerging research direction in this field focuses on extrapolative design—the ability to move beyond the statistical patterns of natural evolution to predict the behavior of synthetic, de novo, or highly mutated sequences.
The Interpolation Trap and the Generalization Gap
Most biological models are trained on databases of natural sequences, which represent only a tiny, evolutionarily "successful" fraction of the astronomical sequence space. When a model encounters a sequence from the "dark proteome" or a heavily engineered variant, its performance often plummets.
Antoine Delaunay and colleagues demonstrated this "generalization gap" in Assessing data size requirements for training generalizable sequence-based TCR specificity models. They found that current models for T-cell receptor (TCR) binding fail to generalize to novel cancer mutations because the training data is orders of magnitude too small. To achieve true out-of-distribution (OOD) generalization, we might need up to 100 million unique training examples—far more than currently exist in public repositories.
This gap is not just a matter of data volume; it is a structural limitation. As Clara Fannjiang and Kadina E. Johnston argue in Machine Learning for Protein Engineering, models often rely on local statistical correlations rather than the underlying biophysical principles that govern how a protein actually folds and functions.
Hallucinating the Missing Evolutionary Context
One way to bridge this gap is to "hallucinate" the missing data. If a protein is an "orphan" (meaning it has no known evolutionary relatives), standard tools like AlphaFold struggle.
Researchers like Le Zhang have addressed this by developing MSA-Generator, a model that generates virtual Multiple Sequence Alignments. By creating high-quality synthetic "relatives," these models provide the structural context necessary for accurate prediction. Similarly, Emanuele Loffredo and Simona Cocco use generative models to restore data balance in TCR datasets, hallucinating synthetic binding sequences to train more robust classifiers.
Anchoring Statistics in Biophysical Reality
To truly extrapolate, a model must understand why a sequence works, not just that it looks like a natural one. This requires moving from pure machine learning to "physics-informed" architectures.
In Overcoming Extrapolation Challenges of Deep Learning by Incorporating Physics in Protein Sequence-Function Modeling, Shrishti Barethiya and Jianhan Chen show that feeding models "physics-based cheat sheets"—such as Rosetta energy terms—allows them to predict the effects of mutations at sites they have never seen before.
This principle was also central to the DREAM6 Promoter Prediction challenge. Pablo Meyer and Eilon Sharon found that the most accurate models weren't just the most complex ones; they were those that combined black-box machine learning with biophysical insights like nucleosome occupancy and transcription factor competition.
Steering Toward Extreme Fitness
A major goal of protein engineering is to design sequences with "extreme" properties—such as ultra-high stability or activity levels—that exceed anything found in nature. Standard models, however, are biased toward the "average" fitness of their training data.
Mostafa Karimi introduced a solution in Data Distillation for extrapolative protein design through exact preference optimization. By adapting alignment techniques used in Large Language Models (LLMs), Karimi trains models on "triplet" relationships—comparing a sequence against both a better and a worse version. This forces the model to learn the direction of improvement, allowing it to navigate the fitness landscape toward peaks that lie far outside the training distribution.
Benchmarking the Generalization Gap
To track progress, the field has moved toward more rigorous benchmarks that explicitly test extrapolation. Kadina E. Johnston and Ava P. Amini developed FLIP2, which uses "biologically informed splits" to evaluate models. Instead of random testing, FLIP2 asks: can a model trained on single mutations predict the effect of triple mutations? Can it predict fitness at sequence positions it has never seen?
These benchmarks, along with tools like FS-mutant by Mingchen Li, reveal a sobering reality: while complex protein language models (pLMs) are powerful, they often struggle to outperform simple linear models when tasked with true extrapolation. This suggests that the next leap in biological AI will not come from larger models alone, but from architectures that can fundamentally reason about the biophysical laws of life.