Evaluating and improving the predictive accuracy of mixing enthalpies and volumes in disordered alloys from universal pre-trained machine learning potentials

Luis Casillas‐Trujillo, Abhijith S. Parackal, Rickard Armiento, Björn Alling

Abstract

Universal pre-trained machine learned interatomic potentials (UPMLIPs) such as MACE, CHGNET, and M3GNET offer the potential to perform large-scale materials simulations with the accuracy of ab-initio methods at a fraction of the computational cost. However, their reliability in high-precision thermodynamic applications, such as predicting the stability of disordered alloys, remains to be fully validated. In this work, we evaluate these models against Density Functional Theory (DFT) calculations for 21 binary alloy systems. We find that while UPMLIPs struggle to achieve the "chemical accuracy" required to consistently predict the correct sign of mixing enthalpies out-of-the-box, they serve as excellent structural pre-relaxers and can be significantly improved through targeted fine-tuning.

Introduction: The Promise of Universal Potentials

In alloy theory, the mixing enthalpy ($\Delta H_{mix}$) is a critical parameter that determines whether a binary system will form a solid solution or phase-separate. Traditionally, calculating these values requires DFT, which is computationally expensive and limits the size of the systems that can be studied. The emergence of "universal" potentials—models pre-trained on massive datasets like the Materials Project—promises a shortcut. These models are designed to be used "off-the-shelf" across a wide range of chemistries. This study investigates whether these models can handle the subtle energy differences (often < 50 meV/atom) that govern alloy thermodynamics.

Benchmarking Performance Across Binary Alloys

We benchmarked MACE, CHGNET, and M3GNET against high-fidelity DFT calculations (PBE and LDA functionals) using Special Quasirandom Structures (SQS) to represent disordered states. The evaluation spanned 21 systems at various concentrations.

Comparison of total energies and mixing enthalpies across 12 binary alloy systems using DFT and three UPMLIPs.

As shown in the broad benchmarking results, while the models generally capture the overall trend of total energy (left columns), the derived mixing enthalpies (right columns) show significant discrepancies. The Mean Absolute Errors (MAEs) for mixing enthalpy at a 50/50 concentration ranged from 27.7 to 41.5 meV/atom. For context, the difference between two different DFT functionals (LDA and PBE) is typically only ~9 meV/atom, meaning the ML error is often larger than the physical signal being measured.

The Precision Problem: Sign Errors and Stability

A major finding is the "MAE Trap": because mixing enthalpies are often small, an error of 30 meV/atom can lead a model to predict the wrong sign for the energy. This results in a qualitative failure where a model predicts an alloy is stable when it is actually unstable, or vice versa.

Mixing enthalpy of Cu-Rh, showing models predicting the wrong sign compared to DFT.

In the Cu-Rh system, for example, DFT (PBE) predicts a positive mixing enthalpy, indicating a tendency toward phase separation. However, both CHGNET and M3GNET predict negative values, incorrectly suggesting a stable solid solution. This inconsistency highlights that "universal" does not necessarily mean "ready-to-use" for high-precision thermodynamics without further refinement.

Accelerating Discovery with Hybrid Workflows

Despite the inaccuracies in absolute energy, the models proved highly effective at predicting forces. They can relax atomic structures into the correct geometries almost as accurately as DFT. This leads to a powerful hybrid workflow: using a UPMLIP like MACE to perform the expensive structural relaxation, followed by a single "static" DFT calculation to obtain the final high-precision energy. This approach yields results nearly identical to full DFT relaxation at a significantly reduced computational cost.

Furthermore, we demonstrated that small-scale fine-tuning can correct these errors. For the Al-Zn system, adding just 34 relevant structures to the training set allowed MACE to correctly predict the mixing behavior, provided the training data settings (like pseudopotentials) perfectly matched the original pre-training environment.

Conclusion

While current UPMLIPs lack the out-of-the-box chemical accuracy required for reliable alloy thermodynamics, they are invaluable tools for accelerating traditional workflows. Future foundation models for materials science should incorporate more diverse data, including disordered solid solutions and liquid phases, to bridge the gap between general chemistry and thermodynamic precision. For now, researchers should treat these models as sophisticated starting points rather than total replacements for ab-initio calculations.

Summary updated

Abstract

The advent of machine learning in materials science opens the way for exciting and ambitious simulations of large systems and long time scales with the accuracy of ab-initio calculations. Recently, several pre-trained universal machine learned interatomic potentials (UPMLIPs) have been published, i.e., potentials distributed with a single set of weights trained to target systems across a very wide range of chemistries and atomic arrangements. These potentials raise the hope of reducing the computational cost and methodological complexity of performing simulations compared to models that require for-purpose training. However, the application of these models needs critical evaluation to assess their usability across material types and properties. In this work, we investigate the application of the following UPMLIPs: MACE, CHGNET, and M3GNET to the context of alloy theory. We calculate the mixing enthalpies and volumes of 21 binary alloy systems and compare the results with DFT calculations to assess the performance of these potentials over different properties and types of materials. We find that small relative energies necessary to correctly predict mixing energies are generally not reproduced by these methods with sufficient accuracy to describe correct mixing behaviors. However, the performance can be significantly improved by supplementing the training data with relevant training data. The potentials can also be used to partially accelerate these calculations by replacing the ab-initio structural relaxation step.
BibTeX
@inproceedings{casillastrujillo2024evaluating,
  author = {Luis Casillas‐Trujillo and Abhijith S. Parackal and Rickard Armiento and Björn Alling},
  title = {{Evaluating and improving the predictive accuracy of mixing enthalpies and volumes in disordered alloys from universal pre-trained machine learning potentials}},
  booktitle = {Physical},
  year = {2024},
  doi = {10.1103/physrevmaterials.8.113803},
}
Figure