Lifelong Machine Learning Potentials

Marco Eckhoff, Markus Reiher

Introduction

Machine Learning Potentials (MLPs) have emerged as a transformative tool in computational chemistry, bridging the gap between the high accuracy of quantum mechanics (QM) and the computational efficiency of classical force fields. Despite their success, traditional MLPs are typically "static" models. They are trained on a fixed dataset and struggle to incorporate new chemical information without undergoing a complete and computationally expensive retraining process. Furthermore, as the chemical complexity of a system increases, the descriptors used to represent atomic environments often suffer from a combinatorial explosion in dimensionality.

Researchers at ETH Zürich have proposed a framework for Lifelong Machine Learning Potentials (lMLPs) designed to overcome these limitations. By integrating novel structural descriptors, specialized optimization techniques, and adaptive data management, the lMLP framework enables models to learn continuously from new data while avoiding the "catastrophic forgetting" that typically plagues neural networks.

The Challenges of Static Potentials

The development of universal MLPs faces two primary hurdles:

  1. The Retraining Tax: When a model encounters a new chemical species or reaction pathway, adding this knowledge usually requires retraining the entire potential on the combined set of old and new data. Failure to do so leads to catastrophic forgetting, where the model loses its ability to accurately predict previously learned systems.
  2. The Elemental Explosion: Standard structural descriptors, such as Atom-Centered Symmetry Functions (ACSFs), scale poorly with the number of chemical elements. Because they require unique functions for every combination of elements, the descriptor vector size grows combinatorially, making simulations of complex, multi-element systems computationally prohibitive.

The lMLP Framework

The proposed lMLP framework introduces three core innovations to facilitate continuous learning and efficient scaling.

1. Element-Embracing Descriptors (eeACSFs)

To solve the scaling issue, the researchers developed element-embracing atom-centered symmetry functions (eeACSFs). Unlike traditional ACSFs, which treat each element as a discrete category, eeACSFs incorporate continuous physical properties from the periodic table, specifically period and group numbers. This allows a fixed-size descriptor vector to represent an arbitrary number of elements. By embedding chemical intuition into the descriptor, the model can generalize across similar elements (e.g., recognizing similarities between different halogens) and maintain a constant computational cost regardless of chemical diversity.

2. The CoRe Optimizer

Learning new data without overwriting old knowledge requires a delicate balance between plasticity and stability. The Continual Resilient (CoRe) optimizer was developed to manage this trade-off. CoRe is a first-order gradient-based optimizer that combines the sign-based robustness of RPROP with the performance of Adam. It introduces "plasticity factors"—a concept inspired by synaptic intelligence in neuroscience—which identify and "freeze" weights that are critical for maintaining performance on older data, preventing them from being significantly altered by new, potentially noisy information.

3. Lifelong Adaptive Data Selection and Uncertainty

The framework includes an automated data management system that identifies redundant or erroneous data points. By pruning the training set, the system ensures that the model rehearses a representative subset of "old" data to maintain memory while focusing on informative "new" data. To determine when new data is needed, the system utilizes an ensemble of neural networks. High variance in predictions among ensemble members serves as a proxy for uncertainty, signaling that the model has encountered an unfamiliar chemical configuration.

Experimental Results

The lMLP framework was evaluated using gas-phase $S_N2$ reaction datasets involving up to 10 different chemical elements (H, C, N, O, F, P, S, Cl, Br, I). The results demonstrated several key advantages:

  • Enhanced Accuracy: The CoRe optimizer achieved a 33% improvement in the accuracy of both energy and force predictions compared to the standard Adam optimizer.
  • Computational Efficiency: For a 10-element system, the eeACSF descriptor required only 153 neurons, whereas standard ACSFs would have required 750. This efficiency becomes even more pronounced as more elements are added to the periodic scope.
  • Robust Data Management: The adaptive selection algorithm successfully reduced the training set to 29% of its original size without compromising accuracy, effectively filtering out outliers and redundant information.
  • Successful Expansion: The model demonstrated the ability to learn new chemical environments (e.g., transitioning from methyl to tert-butyl centers) with minimal loss in performance compared to models trained on the full dataset from scratch.

Discussion and Future Outlook

The lMLP framework represents a significant step toward universal, adaptable potentials. By treating machine learning as a continuous process rather than a one-time training event, researchers can build models that grow in capability alongside their experimental or high-level theoretical data collections.

Currently, the framework is based on second-generation High-Dimensional Neural Network Potentials (HDNNPs), which focus on local atomic environments. Future developments may include extending the framework to 4th-generation potentials to account for long-range electrostatics and non-local interactions. Additionally, exploring architectures that can dynamically "grow" neurons as chemical complexity increases could further enhance the model's capacity for lifelong learning. This approach paves the way for truly universal potentials capable of simulating the vast and diverse landscape of the periodic table.

Summary updated

BibTeX
@article{eckhoff2023lifelong,
  author = {Marco Eckhoff and Markus Reiher},
  title = {{Lifelong Machine Learning Potentials}},
  journal = {Journal of Chemical Theory and Computation},
  year = {2023},
  doi = {10.1021/acs.jctc.3c00279},
}
Figure