Machine learning models are exceptionally good at finding patterns within the boundaries of their training data. However, scientific discovery is fundamentally an act of exploration, not interpolation. If we want to discover a breakthrough catalyst, a high-efficiency thermoelectric material, or a novel drug scaffold, we are looking for global optima that lie far outside known distributions.
As Tomer Weiss et al. observe in Guided diffusion for inverse molecular design, most generative models are restricted to interpolating within the range of their training labels. This creates an "extrapolation gap" where models fail to predict or generate properties that exceed the best examples they have already seen, a limitation echoed by Alexia Jolicoeur-Martineau et al. in Any-Property-Conditional Molecule Generation with Self-Criticism using Spanning Trees. To make machine learning a reliable engine for discovery, we must understand why models fail when they leave familiar territory and how we can design architectures to cross this gap.
The Dimensions of Out-of-Distribution Failure
Why exactly do models break down when they leave familiar territory? The literature identifies two primary failure modes: structural shifts and configurational shifts.
Structural and Compositional Shifts
Models trained on specific chemical families or crystal symmetries struggle when encountering entirely new regions of the periodic table. John A. Keith et al. note in Combining Machine Learning and Computational Chemistry for Predictive Insights Into Chemical Systems that models frequently fail when encountering novel bonding types. Similarly, Raymundo Arroyave argues in Phase Stability Through Machine Learning that the inability to handle out-of-distribution chemical spaces severely limits the discovery of novel materials. When a model encounters an unrepresented element combination or an unfamiliar crystal symmetry, its internal representations lack the necessary context to map these structures to accurate properties.
Configurational and Geometric Shifts
Interatomic potentials and molecular dynamics models are often trained on near-equilibrium bulk structures. When a simulation explores highly distorted geometries, point defects, or high-energy transition states, the underlying physics captured by the model breaks down. Bowen Deng et al. document this in Systematic softening in universal machine learning interatomic potentials, showing that universal potentials systematically underpredict forces in out-of-distribution environments. This matches findings by Silvan Käser et al. in Machine Learning Models of Vibrating H2CO: Comparing Reproducing Kernels, FCHL, and PhysNet, where models lose accuracy on highly distorted molecular geometries because the training data over-samples stable, low-energy configurations.
Why Continuous Latent Spaces Enforce Interpolation
To understand why generative AI struggles to discover truly novel motifs, we must look at how they represent data. Modern deep learning models typically map discrete molecular structures into a continuous latent space. Optimization then occurs by moving along this continuous manifold.
However, as Benjamin Kurt Miller et al. explain in FlowMM: Generating Materials with Riemannian Flow Matching, this design forces the model to interpolate within the training manifold, making it blind to entirely novel structural motifs. When a model samples from a continuous latent space, it naturally blends features of known molecules.
To break free from this geometric constraint, Minghao Guo proposes an alternative in Data-Efficient Graph Grammar Learning for Molecular Generation: instead of continuous latent spaces, models can learn discrete "graph grammars"—production rules that assemble molecules from substructures. By optimizing these discrete assembly rules rather than moving through a continuous space, models can extrapolate to innovative architectures even when trained on highly scarce data.
Benchmarking the Extrapolation Gap
Before we can fix the extrapolation gap, we must be able to measure it. Traditional evaluation metrics evaluate models on random splits of a dataset, which primarily tests interpolation.
To address this, Evan R. Antoniuk et al. introduced BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models, demonstrating that even advanced chemical foundation models experience a tripling of error rates when forced to extrapolate.
To evaluate generative models rather than just predictive ones, Mai Elkady et al. developed a framework called Vertical Validation in Vertical Validation: Evaluating Implicit Generative Models for Graphs on Thin Support Regions. By systematically thinning out specific regions of the training data, they test whether a model can extrapolate into the "thin support" regions where true discovery happens, providing a metric for a model's capacity to generate rare or unprecedented structures.
Strategies for True Extrapolation
How do we build models capable of navigating out-of-distribution chemical space? Researchers are exploring several structural and algorithmic interventions:
- Explicit Inductive Biases: Instead of forcing a model to learn everything from scratch, architectures can be designed to mirror physical realities. For example, Gyoung S. Na et al. show in Predicting thermoelectric properties from chemical formula with explicitly identifying dopant effects that explicitly separating dopants from the host material allows a model to successfully extrapolate to entirely unseen chemical families.
- Multi-View Pre-training: Fusing different representations helps anchor the model when one view becomes unfamiliar. Runhan Shi demonstrates in Prediction of chemical reaction yields with large-scale multi-view pre-training that pre-training on both 1D sequences (SMILES) and 3D spatial geometries significantly improves a model's ability to generalize to out-of-sample reactions involving unseen catalysts.
- Data Densification: When labeled data is sparse, models can fail due to covariate shift. Jina Kim addresses this in Robust Molecular Property Prediction via Densifying Scarce Labeled Data by using a bilevel optimization framework to interpolate sparse training data with abundant unlabeled molecules, smoothing out the transitions into unrepresented chemical neighborhoods.
Key Foundational Work
Can machine learning identify the next high-temperature superconductor? Examining extrapolation performance for materials discovery (2018) — contributed 3 concept elements to this direction.
Hybrid Generative-Discriminative Models for Inverse Materials Design (2018) — contributed 2 concept elements to this direction.
Strategies and Software for Machine Learning Accelerated Discovery in Transition Metal Chemistry (2018) — contributed 1 concept elements to this direction.