Machine Learning Potential Model Based on Ensemble Bispectrum Feature Selection and Its Applicability Analysis

Jiawei Jiang, Li-Chun Xu, Fenglian Li, Jian-Li ShaoMetals 2023 · DOI

TL;DR

This paper introduces a novel two-level ensemble feature selection (EFS) method to reduce the dimensionality of bispectrum descriptors used in machine learning potentials (MLPs) for materials science. By combining data perturbation and ensemble learning strategies, their method significantly cuts down computational costs for training and prediction while maintaining high accuracy, particularly for complex descriptors like qSNAP.

What Problem Does This Paper Address?

Building accurate interatomic machine learning potentials (MLPs) is crucial for simulating material behavior. These MLPs often rely on "descriptors" to represent the local atomic environment. The Spectral Neighbor Analysis Potential (SNAP) uses bispectrum descriptors, which are powerful but can become computationally expensive as their dimensionality increases (controlled by the jmax hyperparameter). This high dimensionality leads to increased complexity in training and prediction without necessarily improving accuracy proportionally. The challenge is to reduce this descriptor dimensionality effectively without sacrificing predictive performance.

Methods

The core of this work is the proposed two-level Ensemble Feature Selection (EFS) method. It operates in two main stages:

  • Level 1: Data Perturbation and Base Feature Selection:
    • The original dataset is perturbed using a bootstrap method to create multiple training subsets.
  • Three distinct base feature selection models are applied to each subset:
  • Gradient Boosting Decision Tree (GBDT): A powerful ensemble method for non-linear relationships.
  • Random Forest (RF): Another ensemble method, known for its robustness.
  • Pearson Correlation Coefficient: A standard method for measuring linear correlation.
  • Each base selector generates a ranked list of feature importances for each subset.
  • Level 2: Ensemble Strategy and Feature Subset Generation:
    • First-level fusion (Rank Product): The feature importance lists from each base selector across all bootstrapped subsets are combined using a Rank Product strategy. This aggregates the importance across different data perturbations.
  • Second-level fusion (Stability Square Weight Rank Product - SSWRP): A novel ensemble strategy is introduced. It combines the feature importances from the first level while weighting them based on the stability of each base feature selector. This means selectors that consistently rank features similarly across different datasets get a higher influence, leading to more robust feature selection.
  • A final ranked feature list is generated, and a feature subset size parameter ($\gamma$) is used to select the top $\gamma \times 100\%$ most important features.

Datasets

The authors evaluated their method on a combination of publicly available and self-built datasets for various metal elements:

  • Self-built datasets: Fe (bcc, fcc, hcp structures)
  • Public datasets: Ni, Cu, Li, Mo, Si, and Ge elements.

These datasets cover a range of sample sizes and feature dimensions, including both standard SNAP features (dimension 56, except Mo) and the higher-dimensional qSNAP features (dimension 1509, except Mo).

Key Insights and Ideas

  • The "curse of dimensionality" in descriptors: Higher jmax values in bispectrum descriptors lead to more complex representations but also a dramatic increase in computational cost.
  • Ensemble learning for robustness: Combining multiple feature selection methods (GBDT, RF, Pearson) and multiple data perturbations (bootstrap) makes the feature selection process more stable and less prone to overfitting.
  • Stability as a key metric: The authors introduce a novel ensemble strategy (SSWRP) that explicitly incorporates feature selection stability into the weighting process, leading to more reliable feature subsets.
  • qSNAP benefits most from feature selection: The higher dimensionality of qSNAP descriptors means they contain more redundant information, making them prime candidates for effective dimensionality reduction via EFS.
  • Trade-off between size and performance: Reducing the feature subset size to around 70% of the original can significantly cut computational costs with minimal impact on prediction accuracy.

Key Results

  • Computational Efficiency:
    • The EFS method reduced training complexity by approximately 51% and prediction complexity by about 30% when using a feature subset size of $\gamma = 0.7$.
  • For qSNAP features, the EFS method achieved an average training time reduction of 43.06%, significantly more effective than for SNAP features.
  • Prediction Accuracy:
    • EFS-qSNAP generally showed improved or comparable prediction accuracy (measured by RMSE) for energy and force compared to original qSNAP features across various metal datasets.
  • While EFS-SNAP sometimes showed a slight decrease in performance compared to SNAP, this was attributed to SNAP's already low dimensionality and lack of redundancy.
  • Stability:
    • The proposed EFS method achieved the highest stability (average 0.94 across datasets) when using a feature subset size of $\gamma = 0.7$, outperforming benchmark methods like t-test, Lasso, RF, GBDT, and Pearson.

Limitations and Future Work

  • Theoretical Foundation: The authors acknowledge that the EFS method lacks a rigorous theoretical derivation based on mathematics and its practicability needs further verification through physical process simulations.
  • Exploration of Relationships: The paper provides a preliminary explanation of feature importance and correlation but suggests further in-depth exploration of these relationships.
  • Further Validation: While the method shows promise, its applicability to a wider range of materials and complex phenomena could be explored.

Summary updated

Abstract

With the continuous improvement of machine learning methods, building the interatomic machine learning potential (MLP) based on the datasets from quantum mechanics calculations has become an effective technical approach to improving the accuracy of classical molecular dynamics simulation. The Spectral Neighbor Analysis Potential (SNAP) is one of the most commonly used machine learning potentials. It uses the bispectrum to encode the local environment of each atom in the lattice. The hyperparameter jmax controls the mapping complexity and precision between the local environment and the bispectrum descriptor. As the hyperparameter jmax increases, the description will become more accurate, but the number of parameters in the bispectrum descriptor will increase dramatically, increasing the computational complexity. In order to reduce the computational complexity without losing the computational accuracy, this paper proposes a two-level ensemble feature selection method (EFS) for a bispectrum descriptor, combining the perturbation method and the feature selector ensemble strategy. Based on the proposed method, the feature subset is selected from the original dataset of the bispectrum descriptor for building the dimension-reduced MLP. As a method application and validation, the data of Fe, Ni, Cu, Li, Mo, Si, and Ge metal elements are used to train the linear regression model based on SNAP for predicting these metals’ atomic energies and forces them to evaluate the performance of the feature subsets. The experimental results show that, compared to the features of SNAP and qSNAP, the training complexity improvement of our EFS method on the qSNAP feature is more effective than SNAP. Compared with the existing methods, when the feature subset size is 0.7 times that of the original features, the proposed EFS method based on the SSWRP ensemble strategy can achieve the best performance in terms of stability, achieving an average stability of 0.94 across all datasets. The training complexity of the linear regression model is reduced by about half, and the prediction complexity is reduced by about 30%.
BibTeX
@inproceedings{jiang2023machine,
  author = {Jiawei Jiang and Li-Chun Xu and Fenglian Li and Jian-Li Shao},
  title = {{Machine Learning Potential Model Based on Ensemble Bispectrum Feature Selection and Its Applicability Analysis}},
  booktitle = {Metals},
  year = {2023},
  doi = {10.3390/met13010169},
}
Figure