Abstract
Large atomic models (LAMs) have undergone remarkable progress recently, emerging as universal or fundamental representations of the potential energy surface defined by the first-principles calculations of atomic systems. However, our understanding of the extent to which these models achieve true universality, as well as their comparative performance across different models, remains limited. This gap is largely due to the lack of comprehensive benchmarks capable of evaluating the effectiveness of LAMs as approximations to the universal potential energy surface. In this study, we introduce LAMBench, a benchmarking system designed to evaluate LAMs in terms of their generalizability, adaptability, and applicability. These attributes are crucial for deploying LAMs as ready-to-use tools across a diverse array of scientific discovery contexts. We benchmark eight state-of-the-art LAMs released prior to April 1, 2025, using LAMBench. Our findings reveal a significant gap between the current LAMs and the ideal universal potential energy surface. They also highlight the need for incorporating cross-domain training data, supporting multi-fidelity modeling, and ensuring the models’ conservativeness and differentiability. As a dynamic and extensible platform, LAMBench is intended to continuously evolve, thereby facilitating the development of robust and generalizable LAMs capable of significantly advancing scientific research.
Introduction
In the same way that Large Language Models (LLMs) have revolutionized natural language processing, Large Atomic Models (LAMs) aim to provide a universal representation of the Potential Energy Surface (PES)—the fundamental "map" that dictates how atoms interact. If perfected, LAMs would allow researchers to simulate any chemical system without needing expensive, slow quantum mechanical calculations.
However, the field currently lacks a unified yardstick. Existing benchmarks are often "siloed," focusing only on specific domains like small molecules or crystals, and rely on static test sets that do not reflect real-world usage, such as long-duration molecular dynamics (MD) simulations. LAMBench addresses this by providing a high-throughput, automated framework to evaluate models on three critical pillars: Generalizability (accuracy on new systems), Adaptability (ease of fine-tuning), and Applicability (speed and stability in production).
The LAMBench Framework
The authors developed the LAMBench-toolkit, an automated workflow that interfaces with the Atomic Simulation Environment (ASE). This toolkit allows for the benchmarking of various LAMs using several novel dimensionless metrics across different task categories:
- Force Field Tasks: Evaluating accuracy in predicting energy, force, and virial tensors across 17 "out-of-distribution" datasets.
- Property Calculation: Testing the ability to derive complex physical properties like phonon frequencies (vibrations in solids) and torsion profiles (twisting in molecules).
- Stability & Efficiency: Measuring throughput (atoms processed per second) and energy drift during 10-picosecond MD runs to ensure simulations do not "explode."
- Multi-task Evaluation: Assessing how models handle different "fidelities" or levels of quantum theory (Exchange-Correlation functionals).
Comprehensive Datasets
To ensure models are not merely memorizing specific domains, LAMBench utilizes a massive variety of data:
- Force Field Evaluation: 17 datasets covering inorganic materials, catalysis, reactions, small molecules, and biomolecules.
- Adaptability: 8 regression tasks from the Matbench suite, including band gaps and formation energies.
- Stability: 9 diverse systems ranging from the Materials Project to the SPICE2 molecular dataset.
Key Insights and Results
Benchmarking eight prominent LAMs (including DPA-2.4, MACE, and SevenNet) revealed several critical findings:
- The "Universal" Gap: Even the top-performing model, DPA-2.4-7M, shows an error metric nearly an order of magnitude higher than an "ideal" model trained specifically on the test data, suggesting we have not yet reached a "GPT-4 level" for atomic physics.
- Domain Overfitting: Most current LAMs are heavily biased toward inorganic materials. Performance often drops significantly when applied to catalysis or biomolecules.
- The XC Mismatch: A major source of error is the mismatch between the quantum theory used for training versus testing. Aligning these through multi-fidelity modeling can reduce errors by half.
- Speed vs. Stability Trade-off: While Orb-v2 achieved the highest inference efficiency, its "non-conservative" nature (predicting forces directly rather than deriving them from energy) led to massive instability in MD simulations. Conversely, conservative models like MatterSim and SevenNet showed near-zero energy drift.
- Pre-training Benefits: Fine-tuning a pre-trained LAM consistently outperforms training from scratch, confirming that these models are learning transferable physical representations.
Conclusion and Future Outlook
LAMBench highlights that while progress is rapid, significant work remains to achieve a truly universal "digital twin" of atomic physics. Future development will focus on expanding the adaptability benchmark to more model architectures and including more complex property calculations, such as ionic diffusion constants. By open-sourcing the toolkit and maintaining an interactive leaderboard, the authors invite the community to contribute new models and datasets to drive the field toward more robust and generalizable atomic representations.
Source Code: github.com/deepmodeling/lambench
Interactive Leaderboard: aissquare.com/openlam?tab=Benchmark
Summary updated