Abstract
We present a scheme to obtain an inexpensive and reliable estimate of the uncertainty associated with the predictions of a machine-learning model of atomic and molecular properties. The scheme is based on resampling, with multiple models being generated based on subsampling of the same training data. The accuracy of the uncertainty prediction can be benchmarked by maximum likelihood estimation, which can also be used to correct for correlations between resampled models and to improve the performance of the uncertainty estimation by a cross-validation procedure. In the case of sparse Gaussian Process Regression models, this resampled estimator can be evaluated at negligible cost. We demonstrate the reliability of these estimates for the prediction of molecular and materials energetics and for the estimation of nuclear chemical shieldings in molecular crystals. Extension to estimate the uncertainty in energy differences, forces, or other correlated predictions is straightforward. This method can be easily applied to other machine-learning schemes and will be beneficial to make data-driven predictions more reliable and to facilitate training-set optimization and active-learning strategies.
Introduction
Machine learning (ML) has revolutionized computational chemistry by enabling the prediction of complex quantum properties—such as molecular energy and atomic forces—at speeds millions of times faster than traditional physics-based simulations. However, a significant hurdle remains: most ML models operate as "black boxes," providing predictions without an inherent measure of confidence.
In chemical research, knowing the uncertainty of a prediction is as important as the prediction itself. It allows researchers to distinguish between safe interpolations within known chemical space and risky extrapolations into unknown territory. While Gaussian Process Regression (GPR) naturally provides uncertainty estimates, these often become unreliable when using "sparse" approximations—a necessary trade-off to handle the large datasets common in modern chemistry. This paper introduces a resampling framework that is both mathematically rigorous and computationally efficient enough for high-throughput applications.
Methodology
The Resampling Framework
The authors propose a sub-sampling approach to build a "committee" of models. Unlike bootstrapping (which samples with replacement), sub-sampling involves training multiple models on different random subsets of the available training data. The "disagreement" or variance between these committee members serves as the raw estimate of the model's uncertainty.
Optimization for Sparse GPR
To ensure the method remains practical, the researchers integrated this committee approach with the "Projected Process" (PP) approximation of GPR. By keeping the "representative set" of atoms consistent across all committee members, the computational cost of evaluating uncertainty for a new molecule becomes effectively zero. During the prediction phase, the process requires only a few additional simple matrix multiplications, removing the "speed penalty" typically associated with ensemble methods.
Calibration via Maximum Likelihood Estimation (MLE)
Raw uncertainty estimates are frequently over- or under-confident. To correct this, the authors use Maximum Likelihood Estimation (MLE) to determine a scaling factor ($v_0$) that aligns predicted uncertainties with observed errors on a validation set. They also introduced a non-linear scaling version to better handle "heavy-tailed" error distributions, which are common when dealing with diverse chemical structures.
Datasets and Benchmarking
The method was rigorously tested on two major chemical datasets:
- CSD-890-R: A collection of organic molecular crystals from the Cambridge Structural Database. The geometries were randomly perturbed ("rattled") to create diverse environments for predicting $^1$H NMR chemical shieldings.
- QM9: A standard benchmark of approximately 134,000 small organic molecules. A subset of 30,000 structures was used to test the prediction of molecular formation energies.
Key Insights
Superior Resolution and Stability
A critical finding was that the sub-sampling estimator provides much better "resolution" than standard GPR variance. It can span two orders of magnitude, making it significantly more effective at identifying truly high-risk predictions. Furthermore, once the MLE calibration is applied, the log-likelihood (a measure of how well the uncertainty matches the error) becomes remarkably stable, regardless of the size of the sub-samples used to build the committee.
The "Internal" Validation Advantage
The authors demonstrated that the model can be calibrated without a separate validation set. Because each sub-sampled model only sees a portion of the data, the "left-out" points can be used to automatically determine the calibration scaling factor. This maximizes the utility of the available training data.
Conclusion
This work provides a robust path forward for making data-driven chemistry more reliable. By providing fast, accurate uncertainty estimates, this method is expected to be a "game-changer" for active learning strategies. In such workflows, the model can autonomously identify areas of high uncertainty and suggest which new experiments or simulations should be performed next to most efficiently improve its own accuracy. While the training phase requires more initial effort to build the committee, the negligible cost during prediction makes it an ideal tool for real-world molecular discovery.
Summary updated