Late-Binding Chemical Identity in Interatomic Potentials

Neural models of source code have a well-documented shortcut problem. They read a variable called sort_array and guess what the code does from the name rather than from its logic. A standard fix is identifier anonymization: every user-defined name becomes a placeholder such as var1 or var2. The same name always maps to the same placeholder inside a function, but the placeholders carry no meaning across functions. This forces the model to learn structure, and on rare or unseen names it sometimes does better than a model given the real names. We propose the same move for machine learning interatomic potentials. These are neural networks trained on quantum-mechanical calculations to predict the energy of a group of atoms and the forces on each atom. Our version withholds the identity of each chemical element from most of the network. Inside one structure, atoms of the same element share a placeholder label, and the labels are shuffled at random between training examples. The true element identities enter only in a small final stage.

Why this matters: "universal" potentials pretrained on materials databases give every element a learned embedding vector, and they are used on the element combinations those databases cover least. Examples are five-to-ten-element high-entropy alloys and crystals generated by swapping one element for another. The hypothesis is that an early, fixed element lookup lets the network memorize which element combinations it has seen, much as code models memorize names. Delaying that lookup ("late binding") should push the network toward geometric and coordination patterns that transfer to unseen combinations. The concrete claim is testable: at matched accuracy on familiar chemistry, a late-binding potential should have measurably lower error on held-out element combinations. The gain should come specifically from placeholders that stay consistent within a structure, not from generic embedding noise.

Core Claim and What Is New

The code literature supplies both the mechanism and its warning label. Empirical Study of Transformers for Source Code trains Transformers with anonymized identifiers and finds they keep most of their ability on structural tasks, occasionally beating full-name models on code completion by treating rare names as generic placeholders. The same study finds anonymization hurts tasks that truly depend on names, such as function naming. VulDeePecker and RealVul use VAR1/var0-style normalization for vulnerability detection. RealVul's ablation credits normalization with F1 gains of up to 14%. Hoppity separates a program's graph structure from its identifier values and resolves the values from local context through a pointer mechanism.

These papers motivate the proposal and supply the method. We borrow two ideas from them: placeholders that keep "same versus different" within one unit while discarding identity across units, and the practice of resolving identity from context at the end rather than looking it up at the start. The function-naming result is a constraint. Element identity is to atoms what names are to function naming: the physics really does depend on it. So full anonymization is ruled out, and identity must re-enter the model somewhere. Our contribution is the specific design that does this (late binding), plus the experiments that isolate whether it helps.

On the materials side, the relevant models all bind identity at the input. Universal potentials such as EquiformerV2, evaluated on metal defects and random alloys in Universal machine learning interatomic potentials poised to supplant DFT in modeling general defects in metals and random alloys, look up a learned embedding per element as the first step. PET-MAD does the same and adds rattled and swapped structures to its training data. Swapping elements in the data is different from what we propose. A swapped structure is a new physical system and needs a new quantum-mechanical calculation. Our method is a constraint on the model's internal representation and needs no new labels. The survey Machine learning for high-entropy alloys and the sparse random-Fourier-feature potential of Dhaliwal et al. both identify the combinatorial growth of element-pair interactions as the bottleneck in many-element systems. That growth is exactly what late binding aims to sidestep. As far as the file shows, no work applies within-structure-consistent species anonymization to interatomic potentials.

Research Program

We build on a compact equivariant message-passing potential of the NequIP or Allegro family. These are graph networks whose internal features rotate consistently with the input structure. We compile them with the kernels of High-performance training and inference for deep equivariant interatomic potentials so that every variant has the same throughput. Forces stay conservative: they are computed as the negative gradient of the energy, because anonymization only changes node input features.

In the late-binding variant, each atom's input feature is a placeholder embedding $p_{\sigma(Z_i)}$. Here $Z_i$ is the atom's element and $\sigma$ is a random injective map from the elements present in the structure to a small pool of $K$ placeholder slots (we use $K = 12$), redrawn for every training example. The first $L$ message-passing layers see only geometry and placeholders. A final binding stage of $M \ll L$ layers then receives the true element embedding $e_{Z_i}$, which it combines with the anonymized features through a gated sum. Its output is the per-atom energy $E_i$, and the total energy is $\sum_i E_i$.

The ratio $L/M$ is the central knob. Setting $M = L$ recovers the standard model. Setting $M = 0$ is full anonymization, which should fail, and confirming that it fails checks the premise. At test time $\sigma$ is drawn randomly, and we report the spread over draws as a stability diagnostic. A well-trained model should have near-zero spread, because the permutation is something it has been trained to ignore.

The program also includes a diagnostic that borrows the code-model evaluation style. For any trained potential, we measure how much the energy of a fixed geometry shifts when chemically similar elements are relabeled (Mo↔W, Nb↔Ta) versus dissimilar ones. We then compare these shifts with the actual quantum-mechanical energy differences. A model whose sensitivity to relabeling is poorly calibrated against physics is, in effect, relying on element names as a shortcut. This diagnostic applies to existing universal potentials whether or not our method works.

Baselines

The naive reference is a standard input-binding model: the identical backbone with element embeddings at the first layer ($M = L$), trained on the same data, with the same optimizer, number of steps, and GPU-hours, and with hyperparameters tuned over the same small grid. Beating it means lower energy and force error on held-out element combinations while staying within noise on the in-distribution split.

The strongest existing reference on crystal stability is the published leaderboard in Matbench Discovery. There, trained on Materials Project and tested on the substitution-generated WBM set, M3GNet reports the best F1 of 0.58 and $R^2$ of 0.59, and MEGNet reports the best discovery acceleration factor (how much better than random the model is at finding stable crystals) of 2.70. Our models train on the same data split, so those numbers are directly comparable. Exceeding M3GNet's F1 with a model of similar size, and exceeding our own input-binding baseline, would count as a win.

For defects and alloys, the strongest reported result is eqV2-31M-omat-mp-salex from Liu et al., with energy RMSE below 5 meV/atom and force RMSE below 100 meV/Å. That model is much larger and trained on OMat24. We therefore treat it as an upper reference rather than a head-to-head competitor, and we make the fair comparison against our own retrained backbone. For the five-element FeNiCuCoCr alloy, the sparse random-Fourier-feature potential of Dhaliwal et al. reports forces within 0.08 eV/Å of the quantum-mechanical reference. We retrain it on our splits as the classical baseline.

Two ablations isolate the key ingredient. The first is embedding masking in the style of BERT: element embeddings are replaced by a single mask token at a matched rate, which injects the same amount of identity noise but breaks within-structure consistency. The second is inconsistent anonymization, where every atom gets its own random placeholder, so the network loses even "these two atoms are the same element."

Why not just…?

Why not just rely on continuous element embeddings, which already share information across similar elements? This is the default, and it is exactly the input-binding baseline. The evidence that it is insufficient is indirect. Liu et al. still find systematic errors on free surfaces and magnetic states. That is not proof of compositional memorization, so the held-out-combination split below is the actual test. If input-binding matches late binding on that split, our hypothesis is wrong, and we will report that.

Why not just add element-swapped structures to the training data, as PET-MAD does? Swapped data requires a new quantum-mechanical calculation for every swap and covers only the swaps someone chose to compute. Our method needs no new labels. The two are also complementary rather than exclusive. We run one arm with a matched budget of extra swapped calculations added to the input-binding baseline. If that arm matches late binding at comparable cost, the practical case for our method weakens, and we will say so.

Why not just apply embedding dropout? This is the most serious threat. Dropout on embeddings is a one-line change and might deliver most of any regularization benefit. The masking ablation is designed to settle it. Our claim holds only if consistent placeholders beat masking at a matched noise rate on held-out combinations. If they do not, the code analogy adds nothing beyond regularization.

Why not just fine-tune the universal model on the target alloy? Fine-tuning needs target data, and the compositional-generalization question is precisely about what happens before any target data exists. We nevertheless report how many target structures each variant needs to reach 100 meV/Å force error. Late binding should need fewer; if it does not, that also counts against it.

Proposed Experiments

The primary experiment is a compositional hold-out. We train on MPtrj with every structure containing chosen element pairs removed: Mo–Ta and Nb–W, plus Co–Cr and Ni–Cu. We then test on the Liu et al. MoNbTaW, Mo25Nb25Ta25W25 defect, and CrCoNi datasets, and on FeNiCuCoCr, which all necessarily contain those pairs. The metrics are energy RMSE, force RMSE, and error on derived properties: stacking-fault energies, and melting ranges compared against the 1500–2000 K reported by Dhaliwal et al.

The decisive pattern has three parts. Late binding should beat input binding on held-out pairs. It should tie on an in-distribution control split. And it should beat masking and inconsistent anonymization. Sweeping $L/M$ should trace a curve that is optimal at an intermediate value. That curve is the materials analogue of Chirkova's finding that structural tasks and name-dependent tasks want different amounts of abstraction.

The secondary experiment is the Matbench Discovery train-on-MP, test-on-WBM protocol. Because WBM is built by elemental substitution, it is a natural test of whether a model has memorized element combinations. We report F1, $R^2$, and discovery acceleration factor against the published M3GNet and MEGNet numbers.

The stability check runs molecular dynamics of the high-entropy alloys at elevated temperature. We verify that late binding does not increase unphysical "structure explosions," and that the spread of predictions across placeholder draws stays below the force error. Finally, the relabeling-sensitivity diagnostic is applied to all trained variants and to the published eqV2 checkpoint. This produces a reusable measurement of how much a universal potential leans on element identity as a shortcut, independent of whether late binding itself wins.