Feature selection for high-dimensional neural network potentials with the adaptive group lasso

Johannes Sandberg, Thomas Voigtmann, Emilie Devijver, Noël JakseMach. Learn. Sci. Technol 2024 · DOI

TL;DR

This paper introduces a novel feature selection method, Adaptive Group Lasso (AGL), for high-dimensional neural network potentials (HDNNPs). By integrating feature selection directly into the training process, AGL effectively reduces the number of atomic features required to accurately describe material properties, leading to more efficient and potentially more stable models.

What Problem Does This Paper Address?

Machine learning interatomic potentials (MLIPs) have revolutionized molecular dynamics simulations by offering a balance between the accuracy of ab initio methods and the computational efficiency of classical force fields. A crucial step in building these MLIPs, particularly High-Dimensional Neural Network Potentials (HDNNPs), is transforming atomic positions into a fixed-size set of "atomic features" or "descriptors." These descriptors capture the local atomic environment and must be invariant to symmetries. However, generating a comprehensive set of descriptors can lead to very high dimensionality, increasing computational cost and potentially hindering model generalization. Traditional feature selection methods often operate independently of the ML model, failing to account for the complex interplay between features and their impact on the model's predictive power. This paper addresses the challenge of efficiently selecting the most relevant atomic features for HDNNPs, aiming to reduce dimensionality without sacrificing accuracy.

Methods

The core contribution of this paper is the development and application of the Adaptive Group Lasso (AGL) method for feature selection in HDNNPs. Unlike traditional Lasso which penalizes individual weights, Group Lasso (GL) penalizes groups of weights, allowing for the simultaneous selection or discarding of entire features. The authors adapt this by using an adaptive penalty, where a regularization parameter is adjusted based on the results of an initial training run with a non-adaptive penalty. This process involves:

  1. Initial Training (Non-adaptive GL): Training the HDNNP with a standard Group Lasso penalty to obtain initial weight estimates.
  2. Adaptive Penalty Construction: Using the weights from the initial training to construct an adaptive regularization parameter for each feature group.
  3. Retraining with Adaptive Penalty: Training the HDNNP again, this time with the adaptive penalty. This step allows features whose weights are deemed less important to be driven to zero.
  4. Feature Freezing and Continued Training: After feature selection, the weights of the selected features are frozen, and the model is retrained without the penalty to recover accuracy lost during the regularization process.

The paper also compares AGL against two common filter methods: Pearson Correlation (PC) and CUR decomposition.

Datasets

The study utilizes datasets for three different systems:

  • Lennard-Jones (LJ) system: A simple, well-known model with perfectly known spherical interactions. Datasets were generated from LAMMPS simulations covering various thermodynamic states (temperature-density phase diagram).
  • Aluminium (Al): A representative sp-bonding metal with predominantly radial interactions. Datasets were derived from DFT-based Ab Initio Molecular Dynamics (AIMD) simulations using VASP, covering solid and liquid phases across a range of temperatures and pressures.
  • Boron (B): A complex system with strongly directional covalent bonding. Datasets were extracted from AIMD trajectories and complemented with additional simulations, including high-pressure conditions, to capture its intricate structure.

Key Insights and Ideas

  • Embedded vs. Filter Methods: The paper strongly advocates for embedded feature selection methods (like AGL) that consider the model's predictive performance, arguing they are superior to filter methods that only analyze feature correlations.
  • Interplay of Features: AGL's ability to account for the interplay between features is crucial, especially in complex systems where features might be individually weak but collectively important.
  • Adaptive Penalty Advantage: The adaptive penalty in AGL is shown to be more effective than a fixed penalty, allowing for more nuanced feature selection.
  • Boron's Complexity: The Boron system highlights the limitations of standard descriptors (Behler-Parrinello SFs) for highly complex materials, suggesting that descriptor choice is as critical as feature selection.
  • Confounding Features: AGL demonstrates robustness against irrelevant (random noise) features, a weakness of filter methods.

Key Results

  • LJ System: AGL successfully reduced the feature set from 22 to 7, achieving comparable accuracy to the full set while significantly improving computational performance. It outperformed PC and CUR in selecting essential features.
  • Aluminium System: AGL selected 10 features, leading to a significant increase in computational speed with only a modest increase in error. AGL, PC, and CUR performed similarly well, likely due to Al's less complex interactions.
  • Boron System: AGL selected 16 features, with a majority being angular, reflecting Boron's directional bonding. While the error increased compared to the full set, computational performance improved. Filter methods (PC and CUR) struggled, with PC models becoming unstable.
  • Computational Speedup: Feature selection generally led to faster simulations, with speedups of up to 5x observed for Al and B.
  • Feature Importance: For LJ and Al, radial features were generally preferred. For Boron, angular features became more prominent due to its directional bonding.

Limitations and Future Work

  • Descriptor Choice: The paper acknowledges that the Behler-Parrinello symmetry functions (SFs) might not be ideally suited for complex systems like Boron, suggesting future work could explore alternative descriptor types.
  • Fitting Forces: The current study focused on fitting energies. Future work should investigate applying AGL when fitting forces, to see how this impacts feature selection.
  • Multicomponent Systems: The authors suggest extending AGL to multicomponent systems, where feature dimensionality can explode combinatorially.
  • Stability: The paper notes recent concerns about MLIP stability and proposes investigating how input dimensionality affects model stability and whether feature selection can alleviate this.
  • Code Availability: The code used in this study is available on GitHub.

Summary updated

BibTeX
@article{sandberg2024feature,
  author = {Johannes Sandberg and Thomas Voigtmann and Emilie Devijver and Noël Jakse},
  title = {{Feature selection for high-dimensional neural network potentials with the adaptive group lasso}},
  journal = {Mach. Learn. Sci. Technol.},
  year = {2024},
  doi = {10.1088/2632-2153/ad450e},
}
Figure