Uncertainty quantification by direct propagation of shallow ensembles

Matthias Kellner, Michele Ceriotti

TL;DR

The authors introduce DPOSE (Direct Propagation of Shallow Ensembles), a lightweight framework for uncertainty quantification that shares most of its neural network weights across ensemble members. It provides a computationally efficient way to estimate errors and, crucially, allows these errors to be accurately propagated to complex derived quantities like atomic forces and thermodynamic averages.

What Problem Does This Paper Address?

Machine learning interatomic potentials (MLIPs) have revolutionized chemistry and materials science by providing quantum-level accuracy at a fraction of the cost. However, these models are often "black boxes." When they encounter a molecular structure far outside their training data (the "extrapolative regime"), they can fail spectacularly without warning.

Uncertainty Quantification (UQ) is the solution, but existing methods face a "trilemma":

  1. Computational Cost: "Deep ensembles" (training many independent models) are accurate but expensive.
  2. Implementation Complexity: Methods like Gaussian Processes are hard to scale to large neural network architectures.
  3. Propagation Issues: Many UQ methods provide an error for a single prediction (e.g., energy), but in physics, we care about derived quantities (e.g., forces, which are derivatives of energy, or heat capacity, which is a fluctuation of energy). Propagating a simple "plus-or-minus" value through these complex functions is mathematically fraught.

Methods

The authors propose DPOSE (Direct Propagation of Shallow Ensembles). The core idea is "shallow" ensembling: instead of training 10 different neural networks, you train one network where every layer is shared except the very last one. This last layer is split into $n_{ens}$ "heads."

  • Training: The model is trained using a Negative Log-Likelihood (NLL) or a Continuously Ranked Probability Score (CRPS) loss. This forces the different heads to not just predict the mean value, but to represent the distribution of possible errors.
  • Direct Propagation: Instead of calculating a single variance and trying to propagate it using calculus (which assumes things are linear and Gaussian), DPOSE propagates the individual predictions of each ensemble head through the entire modeling pipeline. If you need the force, you take the derivative of each head's prediction and then look at their spread.
  • Calibration: They use a post-hoc scaling factor to ensure the predicted uncertainties actually match the observed empirical errors.

Datasets

The authors validated DPOSE across two distinct domains:

  1. General Regression: 10 standard UCI benchmark datasets (e.g., Housing, Concrete, Yacht) to compare against established UQ literature.
  2. Atomistic Modeling: Four high-quality datasets representing different physical challenges:
  3. Liquid Water: High diversity, requires handling long-range interactions.
  4. Lithium Thiophosphate ($Li_3PS_4$): A solid-state electrolyte with amorphous and crystalline phases.
  5. Barium Titanate ($BaTiO_3$): A ferroelectric material with subtle phase transitions.
  6. QM9: A massive dataset of 130,000 small organic molecules.

Key Insights and Ideas

  • Last-Layer Diversity is Enough: Surprisingly, you don't need to ensemble the whole network. Sharing the "feature extraction" layers and only ensembling the final linear mapping provides UQ performance comparable to full deep ensembles at a negligible extra computational cost.
  • The CRPS Advantage: Training on standard NLL loss can sometimes degrade the model's actual prediction accuracy (RMSE). The authors show that using the CRPS loss acts as a "better-behaved" alternative, maintaining high accuracy while producing excellent uncertainty estimates.
  • Handling Correlations Automatically: By propagating individual ensemble members, DPOSE naturally accounts for correlations between different parts of a system. For example, it correctly identifies that errors in a crystal (where all atoms are identical) scale differently than errors in a gas (where atoms are independent).
  • OOD Detection: DPOSE is remarkably effective at flagging "out-of-distribution" structures. In the QM9 dataset, it successfully identified molecules that had undergone "unphysical" rearrangements during optimization, which is vital for active learning loops.

Key Results

  • Efficiency: DPOSE adds almost zero overhead to training and evaluation compared to a standard single model, whereas deep ensembles are $N$ times more expensive.
  • Accuracy: On the UCI benchmarks, DPOSE achieved Relative Log-Likelihood (RLL) scores (a metric where higher is better) that were competitive with or superior to "Mean-Variance Estimation" (MVE) models.
  • Physical Consistency: In water simulations, DPOSE was able to propagate uncertainties to the pair correlation function $g(r)$ and the heat capacity $C_V$, providing physically meaningful confidence intervals that traditional UQ methods struggle to produce.

Limitations and Future Work

  • Underconfidence in Extrapolation: While DPOSE is good at identifying when it's out of its depth, it tends to be "overconfident" (underestimating the true error) in the far-extrapolative regime.
  • Dependency on Calibration: The method relies on a post-hoc calibration step on a validation set. If the validation set isn't representative of the eventual test cases, the UQ might be biased.
  • Future Directions: The authors suggest exploring more sophisticated ways to sample the last-layer weights, perhaps using an approximate Hessian of the loss function to better capture the "shape" of the uncertainty.

Reference: Kellner, M., & Ceriotti, M. (2024). Uncertainty quantification by direct propagation of shallow ensembles. arXiv:2402.16621v2 [physics.chem-ph].

Summary updated

Abstract

Statistical learning algorithms provide a generally-applicable framework to sidestep time-consuming experiments, or accurate physics-based modeling, but they introduce a further source of error on top of the intrinsic limitations of the experimental or theoretical setup. Uncertainty estimation is essential to quantify this error, and make application of data-centric approaches more trustworthy. To ensure that uncertainty quantification is used widely, one should aim for algorithms that are reasonably accurate, but also easy to implement and apply. In particular, including uncertainty quantification on top of an existing architecture should be straightforward, and add minimal computational overhead. Furthermore, it should be easy to manipulate or combine multiple machine-learning predictions, propagating uncertainty over further modeling steps. We compare several well-established uncertainty quantification frameworks against these requirements, and propose a practical approach, which we dub direct propagation of shallow ensembles, that provides a good compromise between ease of use and accuracy. We present benchmarks for generic datasets, and an in-depth study of applications to the field of atomistic machine learning for chemistry and materials. These examples underscore the importance of using a formulation that allows propagating errors without making strong assumptions on the correlations between different predictions of the model.
BibTeX
@article{kellner2024uncertainty,
  author = {Matthias Kellner and Michele Ceriotti},
  title = {{Uncertainty quantification by direct propagation of shallow ensembles}},
  journal = {Mach. Learn. Sci. Technol.},
  year = {2024},
  doi = {10.1088/2632-2153/ad594a},
}
Figure