TL;DR
Researchers from MIT and partners systematically benchmarked several uncertainty quantification (UQ) methods for predicting molecular properties. Their findings reveal a "no free lunch" reality: no single method is universally superior, and performance varies wildly across different chemical datasets, though hybrid approaches like Message Passing Neural Networks combined with Random Forests (MPNN RF) emerge as strong, practical starting points.
What Problem Does This Paper Address?
In drug discovery, we use Quantitative Structure-Activity Relationship (QSAR) models to predict how a new molecule might behave. Because experimental chemistry is expensive and slow, we want models that don't just give a prediction, but also tell us how confident they are. If a model says, "I think this drug is soluble, but I'm only 10% sure," a scientist can choose to skip that experiment and save thousands of dollars.
While many UQ methods exist for neural networks, there hasn't been a rigorous, "apples-to-apples" comparison in the context of molecular regression. This paper fills that gap, testing whether these methods can actually rank predictions by their likely error and provide statistically "honest" confidence intervals.
Methods
The authors evaluated four primary categories of UQ strategies:
- Ensemble-based: Training multiple versions of the same model (via different initializations, bootstrapping data, or "snapshots" during training) and using the variance between their predictions as the uncertainty.
- Mean-Variance Estimation (MVE): Modifying the neural network to output two values—a predicted mean and a predicted variance—and training it using a specialized loss function (Negative Log Likelihood).
- Distance-based: Measuring how "far away" a new molecule is from the training data, either in terms of chemical structure (Tanimoto distance) or the model's internal representation (Latent space distance).
- Union-based (Hybrids): Using a Message Passing Neural Network (MPNN) to learn a molecular representation, then feeding that representation into a "classical" model that handles uncertainty well, like a Gaussian Process (GP) or Random Forest (RF).
Datasets
The study utilized five regression datasets:
- Four Public Benchmarks: Delaney (solubility), Freesolv (solvation energy), Lipo (lipophilicity), and QM7 (atomization energy).
- One Synthetic Dataset (CLogP): Created by the authors to represent a "perfect" scenario with no experimental noise (aleatoric uncertainty), allowing them to isolate how well models handle "I haven't seen this before" (epistemic) uncertainty.
Crucially, they tested both random splits and scaffold splits. Scaffold splits ensure the test molecules are structurally distinct from the training set, mimicking the real-world challenge of discovering entirely new classes of drugs.
Key Insights and Ideas
- The Dataset Dependency Trap: A UQ method that looks like a "gold standard" on one dataset might be the worst performer on another. This suggests that benchmarking UQ on a single dataset is misleading.
- Ranking is Harder than Predicting: Most methods struggled to accurately rank molecules by their absolute error. Even if a model is accurate on average, it often fails to identify exactly which specific predictions are its worst.
- The "Overconfidence" Problem: Traditional ensembling and dropout techniques consistently underestimated uncertainty, especially on out-of-domain (scaffold split) data. They think they know more than they actually do.
- Calibration is a Double-Edged Sword: While "calibrating" a model (adjusting its uncertainty outputs to match observed errors on a validation set) helps on average, it can be highly sensitive to the specific molecules used for that calibration.
Key Results
- Top Performers: Using a modified Wilcoxon Signed-Rank Test to aggregate results, the authors found that MPNN RF, MPNN GP, and MPNN MVE were the most robust across all metrics.
- The Baseline Surprise: Simple Random Forests using static molecular fingerprints (FP RF) performed remarkably well, often rivaling or beating complex neural network UQ methods.
- Metric Matters: The authors used four different metrics (Spearman’s correlation, Miscalibration Area, NLL, and Calibrated NLL). They found that a method might excel at "ranking" errors but fail at providing accurate "confidence intervals," highlighting the need for researchers to choose UQ methods based on their specific end-use case.
Limitations and Future Work
The authors acknowledge that several of these methods (especially ensembling) scale poorly to massive datasets because they require training and running many models. They suggest that "stacking"—combining multiple weak UQ estimators into one meta-estimator—might be a path toward more consistent performance.
Additionally, the study focused on regression (continuous values); future work is needed to see if these trends hold for classification tasks (e.g., "Is this molecule toxic: Yes/No?"). For now, their practical recommendation for researchers is to start with MPNN RF as a reliable first approach for molecular uncertainty.
Summary updated