In standard machine learning, datasets are typically split randomly into training, validation, and test subsets. This practice relies on the assumption that data points are independent and identically distributed (i.i.d.). If you randomly sample images of cats, a model trained on 80% of them will likely generalize well to the remaining 20%.
In the chemical and molecular sciences, however, this i.i.d. assumption breaks down entirely. Molecules are not independent entities randomly scattered across a uniform space; they belong to highly clustered families organized around shared core chemical architectures, known as scaffolds.
When molecular datasets are partitioned randomly, structural analogs—molecules that share the exact same core backbone but differ by only a single atom or functional group—end up distributed across both the training and test sets. This creates a severe evaluation flaw.
The Illusion of High Performance
As Gasser et al. and McLoughlin et al. demonstrate, random data splitting sets a dangerous "leakage trap." Instead of learning the fundamental quantum-mechanical or biochemical principles that govern a molecular property, a model can achieve near-perfect test accuracy simply by memorizing local structural variations around the scaffolds it saw during training.
On paper, the model looks exceptionally capable. But when deployed in a real-world laboratory to evaluate a truly novel compound, its predictive power collapses. Hart et al. emphasize that random train-test splitting fundamentally obscures a model's true ability to generalize to distinct chemical families, leading to over-optimistic performance estimates that fail to translate to actual scientific discovery.
The Mechanism: Partitioning Chemical Space
To simulate how a model will perform when encountering entirely new chemistry, researchers have shifted from random splitting to structural data partitioning. Instead of splitting individual data points, this methodology partitions the underlying chemical space.
Random Splitting (Data Leakage):
Train: [Scaffold A1, Scaffold A2, Scaffold B1, Scaffold C1]
Test: [Scaffold A3, Scaffold B2, Scaffold C2] <-- Model has already seen Scaffolds A, B, and C
Structural Partitioning (True Generalization):
Train: [Scaffold A1, Scaffold A2, Scaffold A3]
Test: [Scaffold B1, Scaffold B2, Scaffold C1] <-- Model must generalize to entirely unseen Scaffolds
There are three primary mechanisms used to achieve this structural separation:
- Scaffold Splitting: Every molecule in the dataset is mathematically reduced to its core topological framework (most commonly using Bemis-Murcko scaffolds). The dataset is then partitioned such that all molecules sharing a specific core backbone are assigned exclusively to one split. If Scaffold A is assigned to the test set, no molecule containing Scaffold A can appear in the training set.
- Cluster-Based Splitting: As implemented by van den Maagdenberg et al., molecules are grouped into clusters based on structural similarity metrics (such as Tanimoto distance over molecular fingerprints) before partitioning. Similarly, Mayr et al. utilized single-linkage clustering to ensure highly similar molecules remain grouped together, forcing the network to bridge the gap between distinct chemical families.
- Subgraph-Constrained Partitioning: To make the evaluation even more stringent, Bushuiev et al. introduced a method that uses the Maximum Common Edge Subgraph distance to guarantee that test samples are structurally distinct from training data. Liu et al. extended this by explicitly excluding structural analogs and near-neighbors from the training distribution entirely, preventing models from relying on local memorization.
Why It Works: Simulating Scaffold Hopping
Enforcing structural partitioning shifts the evaluation of a machine learning model from an in-distribution interpolation task to an out-of-distribution (OOD) extrapolation task.
This directly mirrors the real-world objective of molecular discovery: scaffold hopping. In drug discovery or materials design, scientists rarely want to find a molecule that is a 99% identical copy of an existing compound. Instead, they seek entirely new chemical classes that achieve the same biological or physical effect but possess superior properties, such as lower toxicity or higher stability.
When a model is evaluated on an entirely unseen scaffold, it can no longer rely on simple pattern matching or memorized local features. As Li et al. note, this procedure evaluates a model's true capacity to handle out-of-distribution chemical spaces, which is vital for reliable uncertainty quantification in high-stakes discovery pipelines.
Widespread Application Across Domains
Because structural diversity is a universal challenge in the physical sciences, this partitioning strategy has been adopted across a wide array of domains:
- Graph Learning & Benchmarks: The Open Graph Benchmark by Fey et al. and the DrugOOD curation pipeline replace random splits with scaffold-based partitioning to expose how poorly standard graph neural networks handle distribution shifts. Similarly, WelQrate, developed by Liu et al., uses Bemis-Murcko scaffold splits to establish a gold standard for small-molecule benchmarking.
- ADMET & Toxicity: From predicting human intestinal absorption (Wang et al.) to large-scale absorption, distribution, metabolism, excretion, and toxicity benchmarking in PharmaBench (Niu et al.) and ecotoxicology benchmarks like ADORE (Schür et al.), structural splits prevent over-optimistic performance metrics.
- Physical & Biological Properties: Whether predicting molecular optical peaks (Greenman et al.), drug mechanisms of action (Liu et al.), phenotypic profiles (Moshkov et al.), or training multi-task transfer learning models (Lee et al.), structural partitioning serves as the ultimate reality check.
The philosophy even extends beyond small molecules into materials science. Schopmans et al. demonstrated that when training neural networks on crystal structures, partitioning datasets by unique structure types rather than random configurations forces the model to learn generalized symmetry principles rather than memorizing specific material configurations.
The Remaining Challenge
While structural partitioning provides a much more honest evaluation framework, it also reveals a sobering truth about the current state of molecular AI.
As Deng et al. show in a massive systematic study encompassing over 62,000 models, sophisticated deep learning architectures often struggle significantly more than expected when evaluated on scaffold splits, sometimes failing to outperform traditional machine learning baselines that rely on fixed chemical descriptors. Furthermore, Klarner et al. caution that even standard scaffold splits can sometimes fail to fully simulate the extreme covariate shifts encountered in true discovery environments.
Nevertheless, moving away from random splitting to structural partitioning remains the single most critical step toward building molecular machine learning models that work not just on paper, but in the laboratory.
Key Foundational Work
MoleculeNet: A Benchmark for Molecular Machine Learning (2017) — contributed 2 concept elements to this direction.
DeepTox: Toxicity Prediction using Deep Learning (2016) — contributed 1 concept elements to this direction.
Predicting human intestinal absorption with modified random forest approach: a comprehensive evaluation of molecular representation, unbalanced data, and applicability domain issues (2017) — contributed 1 concept elements to this direction.