In traditional machine learning, we assume that data points are independent and identically distributed (i.i.d.). If you split a dataset of cat photos randomly, a model that learns to recognize a "cat" in the training set should generalize to a "cat" in the test set.
In biology, this assumption fails. Biological sequences—proteins, DNA, and RNA—are the product of billions of years of evolution. They are not independent; they are related through complex lineages. When researchers like Yasser EL‐Manzalawy, Drena Dobbs, and Vasant G Honavar began Evaluating MHC-II Binding Peptide Prediction Methods, they discovered a "memorization wall": models that appeared highly accurate were often just acting as sophisticated lookup tables for evolutionary relatives.
The Problem of Evolutionary Redundancy
The core issue is data leakage. If a training set contains a protein and the test set contains its close evolutionary cousin (a homolog), the model doesn't need to learn the underlying physics of folding or the chemistry of binding. It only needs to recognize the familiar sequence pattern.
As Roshan Rao and colleagues argued in Evaluating Protein Transfer Learning with TAPE, random splitting leads to "inflated performance metrics" because models exploit these hidden relationships. This creates a false sense of security; a model might boast 99% accuracy on a "leaky" benchmark but fail completely when faced with a truly novel protein family in a laboratory setting.
The Mechanism: Homology-Aware Partitioning
To measure true biological understanding, the research community has moved toward homology-aware partitioning. Instead of random splits, data is partitioned based on similarity thresholds.
- Sequence Identity Thresholds: Tools like CD-HIT or MMseqs2 are used to cluster sequences. If two sequences share more than a certain percentage of identity (often 25% or 30%), they are kept in the same partition. This ensures that if a model sees a specific "fold" or "motif" in training, it is never tested on that same motif in the evaluation phase.
- Structural and Fold Clustering: Since structure is more conserved than sequence, researchers like Alex Morehead and Jianlin Cheng in their work on DIPS-Plus advocate for partitioning by structural folds. This forces the model to generalize across entirely different 3D architectures.
- Distance-Based Splits: For highly specific tasks like T-cell receptor (TCR) binding, Filippo Grazioli and Anja Mösch introduced Distance Splits. By using edit distance or 3D RMSD, they ensure the test set is "far" from the training set in biological space.
Why It Works: Learning Principles, Not Patterns
When you remove the "crutch" of homology, the model's performance typically drops—sometimes precipitously. However, the remaining accuracy represents true generalization.
In Detecting sequence signals in targeting peptides using deep learning, José Juan Almagro Armenteros and Ole Winther used stringent homology partitioning to prove that their model, TargetP 2.0, had actually learned biological "zip codes" (sorting signals) rather than just memorizing protein backbones. Similarly, Anton Bushuiev demonstrated in Revealing data leakage in protein interaction benchmarks that many state-of-the-art models for protein-protein interaction (PPI) were failing to learn the physical principles of "stickiness" because they were over-relying on redundant training data.
The Future of Biological Benchmarking
The shift toward rigorous partitioning is redefining what "state-of-the-art" means. It has led to the development of retrieval-augmented models, such as the RAPM framework by Junhui Wu and Zijing Liu, which explicitly separate "memorized" sequence information from "reasoning" capabilities.
By treating sequence similarity as a form of bias to be controlled rather than a feature to be exploited, researchers are finally building models that can navigate the "twilight zone" of low sequence identity—the frontier where the most important biological discoveries are made.