Robustness of Local Predictions in Atomistic Machine Learning Models

Sanggyu Chong, Federico Grasselli, Chiheb Ben Mahmoud, Joe D. Morrow, Volker L. Deringer, Michele Ceriotti

TL;DR

Atomistic machine learning models often predict global properties (like total energy) by summing up contributions from individual atoms. However, because only the total sum is physically observable, the individual "local" predictions can be mathematically arbitrary and unreliable. This paper introduces Local Prediction Rigidity (LPR), a new metric to quantify how robust these local predictions are, and demonstrates that including simple, highly symmetric structures in training sets can "anchor" these predictions, making them far more physically meaningful.

What Problem Does This Paper Address?

In the world of atomistic machine learning (ML), we often use a "divide and conquer" approach. To predict the energy of a large molecule or a complex crystal, models learn to assign an energy value to each atom based on its local neighborhood (environment). While this makes models scalable and interpretable, there is a catch: atomic energies are not physical observables.

Only the total energy of the system can be measured. This creates a "degeneracy" problem: an ML model might correctly predict the total energy while assigning nonsensical values to individual atoms, simply because those errors cancel each other out. This "arbitrariness" makes it dangerous to use local predictions for scientific discovery—such as identifying which specific atoms in a battery electrode are unstable—without knowing if those predictions are robust or just a lucky mathematical fluke.

Methods

The authors propose a quantitative metric called Local Prediction Rigidity (LPR).

Mathematically, LPR is defined as the second derivative of the model's loss function with respect to a perturbation in a local prediction. Think of it like a "spring constant" for a prediction:

  • High LPR: If you try to force a local prediction to change, the model's total error (loss) spikes. This means the local prediction is "rigidly" determined by the training data.
  • Low LPR: You can change the local prediction significantly without much impact on the total error. This indicates the prediction is arbitrary.

The authors derive LPR expressions for:

  1. Linear Models: Where LPR relates directly to the covariance of the feature matrix.
  2. Kernel Methods: Specifically sparse kernel ridge regression.
  3. Neural Networks (NNs): Using automatic differentiation to compute the Hessian of the loss.

Datasets

The study validates LPR across several systems:

  • Numerical Toy Models: 1D polynomial regressions used to demonstrate the core concepts of degeneracy and "anchoring."
  • Amorphous Silicon (a-Si): Used to study how "defective" environments (under- or over-coordinated atoms) affect local stability.
  • Amorphous Carbon (a-C): A complex system with mixed $sp^2$ and $sp^3$ bonding, used to show how different chemical "phases" compete in a global sum.
  • Gallium Arsenide (GaAs): A multi-species system used to explore how the ratio of different elements affects prediction robustness.

Key Insights and Ideas

  • The "Single-Environment" Anchor: The most important finding is that LPR is maximized when the training set includes "single-environment" structures—crystals where every single atom is identical. In these cases, the local prediction is forced to be exactly $Total\_Energy / Number\_of\_Atoms$, removing all arbitrariness.
  • Degeneracy in Disordered Systems: In disordered materials (like glass), the model has many "degrees of freedom" to partition energy. Without symmetric "anchors" in the training set, local predictions in these systems are often highly fragile.
  • LPR vs. Uncertainty: While LPR is related to standard ML uncertainty (like Gaussian Process variance), it specifically captures the robustness of the decomposition. A model might be certain about a global property but completely "floppy" regarding how it distributes that property among atoms.
  • Transferability: Improving LPR doesn't just make the model more interpretable; it often makes it more transferable to new, unseen chemical environments because the underlying "atomic" logic is more physically sound.

Key Results

  • Quantifying Arbitrariness: In amorphous carbon models trained only on mixed structures, LPR values were extremely low ($10^{-10}$ to $10^{-14}$), indicating near-total arbitrariness in how energy was split between $sp^2$ and $sp^3$ atoms.
  • The Power of 1%: By replacing just 1% of a disordered carbon training set with highly symmetric diamond-like and graphite-like structures, the authors increased the LPR of local predictions by orders of magnitude.
  • Reduced Variance: For amorphous silicon, increasing LPR reduced the variance of local energy predictions across a "committee" of 10 different models from ~112 meV to just 3–6 meV per atom.
  • Neural Network Extension: The authors successfully showed that LPR can be applied to modern deep learning architectures, proving that even "flexible" neural networks suffer from—and can be cured of—local prediction instability.

Limitations and Future Work

  • Computational Cost: Calculating the Hessian (required for LPR) can be computationally expensive for very large neural networks or massive datasets.
  • Heuristic Nature: While LPR tells you if a prediction is robust, it doesn't strictly guarantee it is physically correct (though the two are strongly correlated in this context).
  • Scope of Decomposition: This paper focuses on atom-centered decompositions. Future work could extend LPR to other types of partitions, such as bond-centered or multi-body decompositions.
  • Active Learning: The authors suggest that LPR could be used as a criterion in active learning to specifically select new training data that "hardens" the model's internal logic.

Summary updated

Abstract

Machine learning (ML) models for molecules and materials commonly rely on a decomposition of the global target quantity into local, atom-centered contributions. This approach is convenient from a computational perspective, enabling large-scale ML-driven simulations with a linear-scaling cost and also allows for the identification and posthoc interpretation of contributions from individual chemical environments and motifs to complicated macroscopic properties. However, even though practical justifications exist for the local decomposition, only the global quantity is rigorously defined. Thus, when the atom-centered contributions are used, their sensitivity to the training strategy or the model architecture should be carefully considered. To this end, we introduce a quantitative metric, which we call the local prediction rigidity (LPR), that allows one to assess how robust the locally decomposed predictions of ML models are. We investigate the dependence of the LPR on the aspects of model training, particularly the composition of training data set, for a range of different problems from simple toy models to real chemical systems. We present strategies to systematically enhance the LPR, which can be used to improve the robustness, interpretability, and transferability of atomistic ML models.
BibTeX
@article{chong2023robustness,
  author = {Sanggyu Chong and Federico Grasselli and Chiheb Ben Mahmoud and Joe D. Morrow and Volker L. Deringer and Michele Ceriotti},
  title = {{Robustness of Local Predictions in Atomistic Machine Learning Models}},
  journal = {Journal of Chemical Theory and Computation},
  year = {2023},
  doi = {10.1021/acs.jctc.3c00704},
}

Loading related content...

Figure