Out-of-training properties in frozen interatomic potential embeddings are decodable and ordered by physical scale

Natalija Stepurko, Joonatan E. M. LaulainenarXiv 2026 · arXiv

TL;DR

Universal machine-learned interatomic potentials (MLIPs), though trained only on energies and forces, develop internal representations that encode a vast array of "out-of-training" material properties. This paper demonstrates that these frozen embeddings are highly decodable and that information is organized by physical scale across network layers—local structural properties emerge early, while complex electronic properties require deeper integration.

What Problem Does This Paper Address?

In the world of AI for science, "foundation models" for materials—known as Machine-Learned Interatomic Potentials (MLIPs)—are typically trained on a narrow set of targets: the energy, forces, and stresses of atomic systems. While these models are excellent at simulating molecular dynamics, a fundamental question remains: What else do they know?

Training these models is computationally expensive. If their internal "frozen" representations (embeddings) already capture complex chemical and structural nuances, we could potentially reuse them to predict properties like band gaps or bulk moduli without the need for full retraining. The authors set out to systematically probe these latent spaces to understand how information is organized and whether architectural choices (like equivariance) actually matter for downstream tasks.

Methods

The researchers employed a "probing" paradigm, a standard interpretability technique in NLP now applied to materials science.

  1. Model Selection: They compared two state-of-the-art MLIPs with contrasting architectures: ORB-v3 (a non-equivariant message-passing GNN) and UMA-S (an SO(3)-equivariant network).
  2. Embedding Extraction: They froze the networks and extracted activations from every layer, read-out, and equivariant channel.
  3. Supervised Probes: They trained simple "probes"—specifically XGBoost and linear models—to predict nine different material properties from these frozen vectors.
  4. Controls: To isolate the effects of pretraining versus architecture, they used "random-weight" controls (re-initializing models with random weights) and compared results against Magpie, a standard composition-only baseline.
  5. Linear Stitching: They tested whether a linear transformation could map the embeddings of one model onto another while preserving predictive accuracy.
The experimental workflow: from frozen MLIPs to downstream property prediction using various probes and controls.

Datasets

  • Materials Project: The primary dataset consisting of 154,875 crystals and nine target properties (e.g., formation energy, band gap, density, bulk modulus).
  • External Transfer Sets: To test how well these probes generalize, the authors evaluated them on JARVIS-DFT 3D and OQMD, which use different DFT functionals and computational protocols.

Key Insights and Ideas

  • Physical Scale Ordering: One of the most striking findings is that different properties "peak" at different depths in the network. Local, additive properties like density and bulk modulus are well-predicted by early layers. In contrast, electronic properties like band gaps and magnetic labels require the deeper, more integrated representations of later layers.
  • The Value of Pretraining: Pretraining isn't a silver bullet for everything. It provides massive gains for complex properties like "energy above hull" and "band gap," but for simpler properties like density, the architectural prior of the GNN itself (even with random weights) does much of the heavy lifting.
  • Equivariance vs. Representation: While the equivariant UMA-S model significantly outperformed ORB-v3 in classifying crystal systems, the authors found that this wasn't purely due to equivariance. Instead, it was the specific use of rotation-invariant angular-norm features that provided the edge.
  • Latent Smoothness: Even though materials don't form distinct, well-separated clusters in the embedding space based on their properties, a simple nearest-neighbor (k-NN) search often approaches the performance of XGBoost. This suggests the latent space is locally "smooth" and physically meaningful.
Performance of ORB and UMA across different properties. Note the significant gain over the composition-only Magpie baseline.

Key Results

  • Superiority over Baselines: Both MLIPs outperformed the composition-only Magpie baseline on all nine targets, including five properties entirely outside their training objectives.
  • High Predictive Accuracy: At their best layers, the models achieved $R^2$ scores between 0.84 and 0.96 for continuous targets and macro-F1 scores of ~0.90 for categorical targets.
  • Cross-Model Convergence: The "stitching" experiment revealed that despite different architectures, the two models learn functionally similar representations. A linear predictor trained on one model's embeddings can predict properties using the other's embeddings (after a linear map) with minimal loss in accuracy.
Layer-wise performance showing how different properties peak at different depths, reflecting a physical-scale ordering.

Limitations and Future Work

  • Correlation vs. Causality: The study shows that properties are decodable, but it doesn't prove that the MLIPs use this specific information to predict energies and forces.
  • DFT Bias: The probes are trained on DFT labels, meaning they inherit the systematic biases of the underlying physics simulations (e.g., the well-known underestimation of band gaps in GGA).
  • Retrospective Screening: The screening experiments were conducted on known materials. Future work should test these frozen embeddings on truly novel, out-of-distribution candidates.
  • Model Capacity: The authors note that differences in performance between ORB and UMA may be influenced by training data volume and model capacity, not just architecture.

Summary updated

Abstract

Machine-learned interatomic potentials (MLIPs) are trained on energies, forces and stresses, yet their frozen representations can predict other materials properties. We probe ORB-v3 and UMA-S across layers, read-outs and equivariant channels for nine properties of 154,875 Materials Project crystals. Both models outperform composition-only descriptors on all targets, including five outside their training objectives; errors in physical units help assess potential screening applications. Optimal depth varies by property, consistent with a physical-scale ordering. Random-weight controls reveal strongly property-dependent benefits of pretraining. UMA-S's rotation-invariant angular channels improve crystal-system classification, while higher scores from its full tensor disappear under rotation; matched-model controls do not isolate equivariance as the explanation. Accuracy varies across different probes, but nearest-neighbour prediction often approaches XGBoost performance, even though materials do not form distinct clusters according to their properties. We also test whether a linear predictor trained on one model's embeddings can predict properties using the other model's embeddings after a learned linear transformation. Predictive accuracy is largely preserved, despite differences in which atoms the two models place close together. These results guide the reuse of frozen MLIP representations for materials prediction.
BibTeX
@misc{stepurko2026out,
  author = {Natalija Stepurko and Joonatan E. M. Laulainen},
  title = {{Out-of-training properties in frozen interatomic potential embeddings are decodable and ordered by physical scale}},
  year = {2026},
  eprint = {2611.21116},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21116},
}
Figure