A quantitative uncertainty metric controls error in neural network-driven chemical discovery

Jon Paul Janet, Chenru Duan, Tzuhsiung Yang, Aditya Nandy, Heather J. Kulik

Abstract

Machine learning (ML) models, such as artificial neural networks, have emerged as a complement to high-throughput screening, enabling characterization of new compounds in seconds instead of hours. The promise of ML models to enable large-scale chemical space exploration can only be realized if it is straightforward to identify when molecules and materials are outside the model's domain of applicability. Established uncertainty metrics for neural network models are either costly to obtain (e.g., ensemble models) or rely on feature engineering (e.g., feature space distances), and each has limitations in estimating prediction errors for chemical space exploration. We introduce the distance to available data in the latent space of a neural network ML model as a low-cost, quantitative uncertainty metric that works for both inorganic and organic chemistry. The calibrated performance of this approach exceeds widely used uncertainty metrics and is readily applied to models of increasing complexity at no additional cost. Tightening latent distance cutoffs systematically drives down predicted model errors below training errors, thus enabling predictive error control in chemical discovery or identification of useful data points for active learning.

Introduction

The integration of machine learning (ML) into chemical and materials science has drastically accelerated the pace of discovery. By predicting molecular properties in milliseconds, neural networks allow researchers to navigate vast chemical spaces that were previously inaccessible via traditional computational or experimental methods. However, a significant hurdle remains: neural networks are often "overconfident." When presented with a molecule significantly different from their training data—a common occurrence in discovery—they may provide precise but highly inaccurate predictions. In high-stakes environments where experimental validation is costly, a reliable "check engine light" or uncertainty metric is essential.

The Challenge of Uncertainty Quantification

Current methods for quantifying uncertainty in ML models face significant trade-offs. Ensemble models, which involve training multiple versions of the same architecture, are considered the gold standard but are computationally expensive, requiring an order of magnitude more resources. Monte-Carlo dropout offers a cheaper alternative but often underestimates error. Meanwhile, feature space distances—measuring how much an input "looks" like training data based on raw descriptors—require intensive manual feature engineering and may not reflect the features the model actually utilizes for its predictions.

Methodology: Latent Space Distance (LSD)

To address these limitations, we propose Latent Space Distance (LSD). This metric leverages the internal representations learned by the neural network itself. Instead of analyzing raw input features, LSD measures distances in the final hidden layer—the "latent space"—where the network has distilled the input into the most relevant features for the target property.

Comparison of uncertainty quantification methods: feature space distance, latent space distance, Monte-Carlo dropout, and ensemble models.

The implementation of LSD follows a three-step process:

  1. Mapping Training Data: The positions of all training data points are recorded within the model's latent space.
  2. Proximity Measurement: For any new input, the Euclidean distance to the nearest training points in the latent space is calculated.
  3. Calibration: Using a small validation set, these distances are mapped to a Gaussian distribution to provide a concrete error estimate (e.g., ± kcal/mol).

Because the latent space activations are computed during a standard forward pass, LSD adds virtually no additional computational cost to the prediction process.

Experimental Validation

The efficacy of LSD was tested across two distinct chemical domains:

Inorganic Chemistry

Using a dataset of 1,901 transition metal complexes, the model was tested against a challenging "out-of-sample" set of 116 complexes from the Cambridge Structural Database (CSD). These complexes were specifically selected for their chemical distinctness from the training set. While the raw model exhibited a high mean error of 8.6 kcal/mol on this set, applying a latent distance filter to select the "most certain" 25% of molecules reduced the error to a much more usable 3 kcal/mol.

Organic Chemistry

On the QM9 dataset (134,000 small organic molecules), the researchers simulated a discovery scenario by training on only 5% of the data. LSD significantly outperformed ensemble methods in identifying high-accuracy predictions. At a fixed error tolerance, LSD identified a pool of 4,200 reliable predictions, whereas ensemble metrics only identified 74.

Key Insights

A primary advantage of LSD is that the latent space acts as a "smarter" coordinate system. It naturally filters out noise and focuses on the chemical features relevant to the property being predicted. Furthermore, the study revealed a critical "blind spot" in ensemble models: when a molecule is sufficiently far from the training distribution, ensemble members can become "unanimously wrong," showing low variance despite high error. LSD does not suffer from this overconfidence.

By tightening the latent distance cutoff, researchers can systematically drive down prediction errors, sometimes achieving accuracy superior to the model's original training error. This makes LSD an ideal tool for active learning, where the metric can identify the most "useful" (high-uncertainty) data points to label next, maximizing the efficiency of model improvement.

Conclusion

Latent Space Distance provides a robust, low-cost, and scalable solution for uncertainty quantification in chemical AI. By enabling predictive error control and identifying out-of-distribution molecules, it provides the necessary safeguards for large-scale chemical space exploration. Future work will explore non-Euclidean distance metrics and the application of LSD to more complex architectures, such as graph neural networks and image-based models.

Summary updated

Abstract

Machine learning (ML) models, such as artificial neural networks, have emerged as a complement to high-throughput screening, enabling characterization of new compounds in seconds instead of hours. The promise of ML models to enable large-scale chemical space exploration can only be realized if it is straightforward to identify when molecules and materials are outside the model's domain of applicability. Established uncertainty metrics for neural network models are either costly to obtain (<i>e.g.</i>, ensemble models) or rely on feature engineering (<i>e.g.</i>, feature space distances), and each has limitations in estimating prediction errors for chemical space exploration. We introduce the distance to available data in the latent space of a neural network ML model as a low-cost, quantitative uncertainty metric that works for both inorganic and organic chemistry. The calibrated performance of this approach exceeds widely used uncertainty metrics and is readily applied to models of increasing complexity at no additional cost. Tightening latent distance cutoffs systematically drives down predicted model errors below training errors, thus enabling predictive error control in chemical discovery or identification of useful data points for active learning.
BibTeX
@inproceedings{janet2019quantitative,
  author = {Jon Paul Janet and Chenru Duan and Tzuhsiung Yang and Aditya Nandy and Heather J. Kulik},
  title = {{A quantitative uncertainty metric controls error in neural network-driven chemical discovery}},
  booktitle = {Chemical},
  year = {2019},
  doi = {10.1039/c9sc02298h},
}
Figure