Abstract
Uncertainty quantification (UQ) is critical in molecular property prediction for drug discovery, where model confidence guides experimental resource allocation. Despite the proliferation of neural network-based UQ methods, a lack of consensus exists regarding their relative performance. This study systematically benchmarks several UQ strategies across five regression datasets using multiple performance metrics. Our findings indicate that no single method is universally superior, and most struggle to reliably rank errors. However, hybrid approaches, particularly Message Passing Neural Networks combined with Random Forests (MPNN RF), demonstrate robust performance and serve as a practical recommendation for researchers.
Introduction
Quantitative Structure-Activity Relationship (QSAR) models are essential tools in modern drug discovery, enabling the prediction of molecular properties before synthesis. Given the high cost and time-intensive nature of experimental chemistry, it is vital that these models provide not only a prediction but also a reliable measure of confidence. A model capable of signaling "I don't know" allows researchers to prioritize experiments and avoid costly errors. While various UQ techniques have been developed for neural networks, their efficacy in the specific context of molecular regression has remained under-explored.
Methodology
The study evaluates four primary categories of UQ strategies, as illustrated in the figure below.
- Ensemble-based Methods: These involve training multiple instances of the same model architecture using different initializations or data subsets. Uncertainty is derived from the variance among the individual models' predictions.
- Mean-Variance Estimation (MVE): This approach modifies the neural network architecture to output both a predicted mean and a predicted variance, typically trained using a Negative Log Likelihood (NLL) loss function.
- Distance-based Methods: These quantify uncertainty based on the distance of a new molecule from the training set, measured either in chemical structure space or within the model's internal latent representation.
- Union-based (Hybrid) Methods: These leverage the feature-learning capabilities of Message Passing Neural Networks (MPNNs) to generate molecular embeddings, which are then processed by classical models known for robust uncertainty handling, such as Gaussian Processes (GP) or Random Forests (RF).
Experimental Setup
The methods were tested on five regression datasets: Delaney (solubility), Freesolv (solvation energy), Lipo (lipophilicity), QM7 (atomization energy), and a synthetic CLogP dataset designed to isolate epistemic uncertainty. To simulate real-world drug discovery challenges, the authors utilized both random and scaffold splits, the latter ensuring that test molecules are structurally distinct from the training data.
Results and Discussion
The benchmarking revealed several critical insights into the current state of UQ in molecular AI:
- Dataset Sensitivity: Performance was highly dependent on the specific dataset. A method excelling on one benchmark often performed poorly on another, cautioning against drawing broad conclusions from single-dataset evaluations.
- The Ranking Challenge: Most methods struggled to accurately rank predictions by their absolute error. While models might be accurate on average, identifying specific high-error predictions remains difficult.
- Overconfidence in Neural Models: Traditional ensembling and dropout techniques frequently underestimated uncertainty, particularly on out-of-domain scaffold splits.
- Robustness of Hybrids: Aggregated results showed that MPNN RF, MPNN GP, and MPNN MVE were the most consistently robust across metrics such as Spearman’s correlation and Miscalibration Area.
- Strong Baselines: Interestingly, simple Random Forests using static molecular fingerprints (FP RF) often rivaled more complex neural UQ methods in performance.
Conclusion
The study demonstrates that existing UQ methods for molecular property prediction are not yet sufficient for all use cases, particularly regarding error ranking. There is no "free lunch," and the choice of method should be guided by the specific application and metric of interest. However, for practitioners seeking a reliable starting point, the MPNN RF hybrid approach offers a strong balance of predictive power and uncertainty quantification.
Summary updated