Prediction rigidities for data-driven chemistry

Sanggyu Chong, Filippo Bigi, Federico Grasselli, Philip Loche, Matthias Kellner, Michele CeriottiFaraday 2024 · DOI

TL;DR

This paper introduces "prediction rigidities" (PRs), a novel family of metrics derived from the ML model's loss function, to quantify the robustness and interpretability of predictions in data-driven chemistry. PRs offer a powerful tool to understand how ML models learn, guide dataset construction, and assess model behavior at both global and local levels, even for complex coarse-grained systems.

What Problem Does This Paper Address?

Machine learning (ML) models are increasingly used in chemistry to predict properties of molecules and materials. However, understanding how these models arrive at their predictions and ensuring their reliability (interpretability and transferability) remains a significant challenge. This is particularly true when models decompose complex properties into simpler, interpretable components (like atomic energies or contributions from different interactions). The arbitrariness in these decompositions can lead to models that are not robust or transferable. This paper addresses the need for a quantitative tool to assess this robustness and guide the development of more reliable ML models in chemistry.

Methods

The core of this work is the introduction and application of "prediction rigidities" (PRs). PRs are derived from a constrained loss function that measures how much the model's loss changes when a specific prediction is slightly perturbed. This perturbation is mathematically handled using Lagrange multipliers. The key idea is that a "rigid" prediction is one that doesn't change much when slightly altered, indicating robustness.

The authors extend the PR formalism to:

  • Global and Local PRs: PRs can assess the overall model confidence (global) or the confidence in specific local predictions (e.g., atomic environments).
  • Component-wise PRs (CPRs): For models that predict a property as a sum of components (like different interaction types), CPRs can evaluate the robustness of each individual component's prediction.
  • Last-Layer PRs for Neural Networks (NNs): A practical formulation is presented for deep NNs, leveraging the properties of the last layer to make calculations computationally feasible without re-training. This approach is grounded in Laplace approximation and neural tangent kernel theories.
  • Generalized Gauss-Newton Approximation: To avoid computationally expensive second derivatives of the loss function, a pseudo-Hessian approximation is used.

Datasets

The paper utilizes several standard datasets for training and evaluating ML models:

  • QM9 dataset: Used for training Behler-Parrinello (SOAP-BPNN) and PaiNN models to predict molecular total energies.
  • Si10 clusters dataset: Used for training MACE models.
  • GAP-17 carbon dataset: Used for training a linear LE-ACE model.
  • Water dimer configurations: Used for training SOAP+LODE models.
  • Coarse-grained water MD simulations: Used to generate training data for MACE models in a coarse-grained context.

The paper does not introduce new benchmark datasets but rather applies the PR formalism to existing ones to demonstrate its utility.

Key Insights and Ideas

  • PRs as a Universal Metric: Prediction rigidities offer a unified way to assess model robustness across different ML architectures and problem types in chemistry, from small molecules to complex materials and coarse-grained systems.
  • Understanding Model Learning: PRs reveal how models learn by showing which parts of the prediction are stable and which are sensitive to changes, offering insights into the model's internal representation of physics.
  • Guiding Dataset Construction: PRs can be used to identify "uncertain" or "rigid" regions in the data, guiding the selection of new data points for training to improve model accuracy and reduce errors efficiently. This is particularly useful for active learning and fine-tuning.
  • Interpreting Component-wise Predictions: CPRs are crucial for models that decompose properties. They highlight issues like "double counting" or non-orthogonal contributions between different physical interactions (e.g., short-range vs. long-range forces), leading to more interpretable models.
  • Last-Layer Approximation is Valid: For neural networks, focusing on the last layer's latent features to compute PRs is a computationally efficient and accurate approach, validated by strong correlations with empirical errors.
  • Coarse-Graining Robustness: PRs can effectively monitor the convergence and robustness of coarse-grained ML models, providing insights into their ability to capture underlying physics even with simplified representations.

Key Results

  • Correlation with Empirical Errors: The inverse of the last-layer PRs showed strong linear correlations with empirical errors (absolute and squared) for various NN architectures (SOAP-BPNN, PaiNN, MACE), validating PRs as a measure of model uncertainty.
  • Local Prediction Robustness: Last-layer Local Prediction Rigidities (LPRs) were inversely correlated with the variance in local energy predictions, indicating that higher LPR environments are more robust.
  • Dataset Size Impact: PR distributions showed dependence on dataset makeup but were largely insensitive to optimization details (e.g., training trajectory).
  • PR-Guided Augmentation: A PR-guided dataset augmentation strategy significantly reduced RMSE for extending a model to new systems (e.g., surface-containing carbon structures) with fewer added samples compared to random selection.
  • Component Orthogonalization Benefits: Component-wise PRs revealed that orthogonalizing descriptors in models like ACE and SOAP+LODE leads to clearer separation of physical contributions and improved learning behavior, resolving degeneracies.
  • Coarse-Grained Model Assessment: PRs for coarse-grained water models correlated well with the accuracy of pair correlation functions and local order parameter distributions, demonstrating their utility in assessing the robustness of these complex simulations.

Limitations and Future Work

  • Linearity Assumption: The current formalism is primarily applicable to regression models where the prediction is linear with respect to the (last-layer) features.
  • Extension to Non-linear Models: Future work should extend the PR formalism to models with non-linearity in the prediction layer, such as classification models.
  • Calibrated Uncertainty Estimates: While PRs provide a qualitative indicator of convergence and robustness, they do not directly yield calibrated uncertainty estimates, especially in cases like coarse-graining where intrinsic errors exist.
  • Broader Applicability: The authors suggest that PRs can be applied to models trained on experimental data and to a wider range of ML architectures.
  • Active Learning Refinements: Further research could explore more sophisticated methods for imposing displacements on buffer atoms in active learning strategies (e.g., constrained MD).

Summary updated

Abstract

The widespread application of machine learning (ML) to the chemical sciences is making it very important to understand how the ML models learn to correlate chemical structures with their properties, and what can be done to improve the training efficiency whilst guaranteeing interpretability and transferability. In this work, we demonstrate the wide utility of prediction rigidities, a family of metrics derived from the loss function, in understanding the robustness of ML model predictions. We show that the prediction rigidities allow the assessment of the model not only at the global level, but also on the local or the component-wise level at which the intermediate (<i>e.g.</i> atomic, body-ordered, or range-separated) predictions are made. We leverage these metrics to understand the learning behavior of different ML models, and to guide efficient dataset construction for model training. We finally implement the formalism for a ML model targeting a coarse-grained system to demonstrate the applicability of the prediction rigidities to an even broader class of atomistic modeling problems.
BibTeX
@inproceedings{chong2024prediction,
  author = {Sanggyu Chong and Filippo Bigi and Federico Grasselli and Philip Loche and Matthias Kellner and Michele Ceriotti},
  title = {{Prediction rigidities for data-driven chemistry}},
  booktitle = {Faraday},
  year = {2024},
  doi = {10.1039/d4fd00101j},
}
Figure