When training machine learning models on text or images, we typically rely on a foundational assumption: our data points are independent and identically distributed (i.e.d.). If we randomly partition a dataset into training, validation, and test sets, the performance on the test set serves as a reliable proxy for how well the model will generalize to the real world.
In biology, this assumption breaks down entirely.
Because all living organisms share a common evolutionary history, biological sequences—whether proteins, RNA, or DNA—are fundamentally dependent on one another. Through the process of evolution, nature copies, modifies, and replicates sequence motifs across different species and functional families. When machine learning practitioners apply standard random data splits to biological sequences, they inadvertently introduce sequence redundancy bias. This creates a severe form of data leakage where closely related evolutionary variants (homologs) end up in both the training and testing sets, turning a generalization challenge into a simple exercise in memorization.
The Illusion of Generalization
Why does sequence redundancy matter so much? Consider a model trained to predict protein-protein interactions (PPI) or binding affinities. If the training set contains a specific protein sequence and the test set contains an evolutionary relative that shares 80% sequence identity, the model does not need to understand the underlying biophysical or chemical rules of interaction to make a correct prediction. It simply needs to recognize the identity of the protein and recall its associated label.
As François Charih et al. observe in Sequence-based protein-protein interaction prediction and its applications in drug discovery, high sequence identity between training and evaluation sets allows models to act as sophisticated lookup tables. This inflation of performance metrics masks a critical vulnerability: when deployed in a real-world laboratory setting to discover entirely new protein families or de novo designs, the model's accuracy collapses.
This phenomenon is often described as hitting a "memorization wall." For instance, Filippo Grazioli et al. demonstrated in On TCR binding predictors failing to generalize to unseen peptides that deep learning models evaluating T-cell receptor (TCR) binding achieve stellar performance on standard random benchmarks but drop to near random chance when evaluated on unseen peptides. The models were not learning the physical principles of immune recognition; they were memorizing recurring short motifs present across the leaky train-test splits.
How Random Splits Cause Data Leakage
To understand why random splitting fails, we must look at how evolutionary relationships structure biological data. If we visualize the space of a protein family, it resembles a hierarchical tree rather than a uniform cloud of points.
Random Splitting (Leaky):
Train Branch A: ─── Q1 ─── Q2 ─── (Model memorizes Branch A features)
Test Branch A: ─── Q3 ─── (Easy lookup, inflated accuracy)
Homology-Aware Splitting (Rigorous):
Train Branch A: ─── Q1 ─── Q2 ─── Q3
Test Branch B: ─── R1 ─── R2 ─── (Forces learning of fundamental biology)
When we sample randomly from this tree, we pick leaves from the same sub-branches for both training and testing. The model exploits these local site-specific signals and overrepresented short motifs. Jiahui Zhang et al. highlight this in PepBenchmark: A Standardized Benchmark for Peptide Machine Learning, noting that without rigorous deduplication, performance metrics are artificially driven up by overrepresented sequence fragments rather than true biological or chemical understanding.
The consequences of this bias span across multiple biological domains:
- Genomics: In mRNA property prediction, Ruian Shi et al. show in mRNABench: A Curated Benchmark for Mature mRNA Property and Function Prediction that random partitioning allows models to exploit sequence similarities between related genes, making it difficult to determine if a model has grasped fundamental regulatory rules.
- Protein Engineering: When searching for optimal natural starting points within a protein family, Peter Mørch Groth argues in FLOP: Tasks for Fitness Landscapes Of Protein wildtypes that random splits mislead researchers with overly optimistic estimates, failing to reflect a model's true capacity to extrapolate to novel, distantly related wildtypes.
- Non-coding RNA: In microRNA identification, Bin Liu et al. found in Identification of Real MicroRNA Precursors with a Pseudo Structure Status Composition Approach that genomic datasets frequently suffer from homology bias, where high sequence identity allows models to bypass structural principles in favor of simple pattern matching.
Designing Homology-Aware Splits
To force machine learning models to learn generalizable biophysical properties rather than memorizing evolutionary lineages, the community has shifted toward homology-aware data partitioning.
The core mechanism is straightforward: instead of partitioning individual samples randomly, we cluster sequences based on their evolutionary similarity and assign entire clusters to either the training or testing set. This ensures that no sequence in the test set shares a significant degree of similarity with any sequence in the training set.
1. Sequence Identity Thresholds
The most common approach is to enforce a strict sequence identity cutoff using alignment tools like MMseqs2 or CD-HIT. For example, in Evaluating Protein Transfer Learning with TAPE, Roshan Rao et al. structured benchmarks by filtering training and test sets by a strict 25% sequence identity threshold. This "twilight zone" of sequence similarity ensures that the evaluation reflects the model's ability to generalize to entirely new protein families. Similarly, Bin Liu et al. utilized a strict sequence similarity cutoff in iDNA-Prot|dis: Identifying DNA-Binding Proteins by Incorporating Amino Acid Distance-Pairs and Reduced Alphabet Profile into the General Pseudo Amino Acid Composition to emphasize the discovery of structural patterns over simple sequence memorization.
2. Ligand-Disjoint and Cross-Family Splits
For interaction tasks, partitioning must happen on both sides of the interaction. In TCR or drug-target predictions, the authors of On TCR binding predictors failing to generalize to unseen peptides advocate for ligand-disjoint data partitioning, where specific target molecules are completely absent from the training set. For broader structural tasks, Kanti V. Mardia et al. relied on the SABmark dataset in A generative, probabilistic model of local protein structure, which is explicitly curated for low sequence similarity within structural folds to ensure the learning of generalizable physical principles.
3. Hybrid and Motif-Aware Clustering
As models grow more sophisticated, simple global sequence alignments may miss local similarities that still allow for data leakage. To counter this, Jiahui Zhang et al. implemented a hybrid partitioning strategy in PepBenchmark that clusters samples based on both global sequence similarity and local k-mer motif overlap. This multi-layered defense prevents models from exploiting short, recurring functional fragments.
Why Homology-Aware Splitting Works
Enforcing homology-aware splits fundamentally changes what a neural network optimizes for during training. When evolutionary relatives are stripped from the training data, the model can no longer minimize its loss function via memorization.
Faced with a test set composed of entirely novel evolutionary branches, the model is forced to capture deeper, invariant features of the sequence space—such as secondary structure, electrostatic fields, and long-range biophysical constraints.
While switching to homology-aware evaluation often results in a significant drop in reported validation scores, it provides an honest, unvarnished metric of a model's true utility. By eliminating sequence redundancy bias, we bridge the gap between computational evaluation and real-world biological discovery.