Uncertainty Quantification Using Neural Networks for Molecular Property Prediction

Lior Hirschfeld, Kyle Swanson, Kevin Yang, Regina Barzilay, Connor W. Coley

TL;DR

Researchers from MIT and partners systematically benchmarked several uncertainty quantification (UQ) methods for predicting molecular properties. Their findings reveal a "no free lunch" reality: no single method is universally superior, and performance varies wildly across different chemical datasets, though hybrid approaches like Message Passing Neural Networks combined with Random Forests (MPNN RF) emerge as strong, practical starting points.

What Problem Does This Paper Address?

In drug discovery, we use Quantitative Structure-Activity Relationship (QSAR) models to predict how a new molecule might behave. Because experimental chemistry is expensive and slow, we want models that don't just give a prediction, but also tell us how confident they are. If a model says, "I think this drug is soluble, but I'm only 10% sure," a scientist can choose to skip that experiment and save thousands of dollars.

While many UQ methods exist for neural networks, there hasn't been a rigorous, "apples-to-apples" comparison in the context of molecular regression. This paper fills that gap, testing whether these methods can actually rank predictions by their likely error and provide statistically "honest" confidence intervals.

Methods

The authors evaluated four primary categories of UQ strategies:

  1. Ensemble-based: Training multiple versions of the same model (via different initializations, bootstrapping data, or "snapshots" during training) and using the variance between their predictions as the uncertainty.
  2. Mean-Variance Estimation (MVE): Modifying the neural network to output two values—a predicted mean and a predicted variance—and training it using a specialized loss function (Negative Log Likelihood).
  3. Distance-based: Measuring how "far away" a new molecule is from the training data, either in terms of chemical structure (Tanimoto distance) or the model's internal representation (Latent space distance).
  4. Union-based (Hybrids): Using a Message Passing Neural Network (MPNN) to learn a molecular representation, then feeding that representation into a "classical" model that handles uncertainty well, like a Gaussian Process (GP) or Random Forest (RF).

Datasets

The study utilized five regression datasets:

  • Four Public Benchmarks: Delaney (solubility), Freesolv (solvation energy), Lipo (lipophilicity), and QM7 (atomization energy).
  • One Synthetic Dataset (CLogP): Created by the authors to represent a "perfect" scenario with no experimental noise (aleatoric uncertainty), allowing them to isolate how well models handle "I haven't seen this before" (epistemic) uncertainty.

Crucially, they tested both random splits and scaffold splits. Scaffold splits ensure the test molecules are structurally distinct from the training set, mimicking the real-world challenge of discovering entirely new classes of drugs.

Key Insights and Ideas

  • The Dataset Dependency Trap: A UQ method that looks like a "gold standard" on one dataset might be the worst performer on another. This suggests that benchmarking UQ on a single dataset is misleading.
  • Ranking is Harder than Predicting: Most methods struggled to accurately rank molecules by their absolute error. Even if a model is accurate on average, it often fails to identify exactly which specific predictions are its worst.
  • The "Overconfidence" Problem: Traditional ensembling and dropout techniques consistently underestimated uncertainty, especially on out-of-domain (scaffold split) data. They think they know more than they actually do.
  • Calibration is a Double-Edged Sword: While "calibrating" a model (adjusting its uncertainty outputs to match observed errors on a validation set) helps on average, it can be highly sensitive to the specific molecules used for that calibration.

Key Results

  • Top Performers: Using a modified Wilcoxon Signed-Rank Test to aggregate results, the authors found that MPNN RF, MPNN GP, and MPNN MVE were the most robust across all metrics.
  • The Baseline Surprise: Simple Random Forests using static molecular fingerprints (FP RF) performed remarkably well, often rivaling or beating complex neural network UQ methods.
  • Metric Matters: The authors used four different metrics (Spearman’s correlation, Miscalibration Area, NLL, and Calibrated NLL). They found that a method might excel at "ranking" errors but fail at providing accurate "confidence intervals," highlighting the need for researchers to choose UQ methods based on their specific end-use case.

Limitations and Future Work

The authors acknowledge that several of these methods (especially ensembling) scale poorly to massive datasets because they require training and running many models. They suggest that "stacking"—combining multiple weak UQ estimators into one meta-estimator—might be a path toward more consistent performance.

Additionally, the study focused on regression (continuous values); future work is needed to see if these trends hold for classification tasks (e.g., "Is this molecule toxic: Yes/No?"). For now, their practical recommendation for researchers is to start with MPNN RF as a reliable first approach for molecular uncertainty.

Summary updated

Abstract

Uncertainty quantification (UQ) is an important component of molecular property prediction, particularly for drug discovery applications where model predictions direct experimental design and where unanticipated imprecision wastes valuable time and resources. The need for UQ is especially acute for neural models, which are becoming increasingly standard yet are challenging to interpret. While several approaches to UQ have been proposed in the literature, there is no clear consensus on the comparative performance of these models. In this paper, we study this question in the context of regression tasks. We systematically evaluate several methods on five regression data sets using multiple complementary performance metrics. Our experiments show that none of the methods we tested is unequivocally superior to all others, and none produces a particularly reliable ranking of errors across multiple data sets. While we believe that these results show that existing UQ methods are not sufficient for all common use cases and further research is needed, we conclude with a practical recommendation as to which existing techniques seem to perform well relative to others.
BibTeX
@article{hirschfeld2020uncertainty,
  author = {Lior Hirschfeld and Kyle Swanson and Kevin Yang and Regina Barzilay and Connor W. Coley},
  title = {{Uncertainty Quantification Using Neural Networks for Molecular Property Prediction}},
  journal = {Journal of Chemical Information and Modeling},
  year = {2020},
  url = {https://openreview.net/forum?id=wcIBeD0n9O},
}

Loading related content...

Figure