Masked Wyckoff Models for Context-Aware Element Substitution

Most new stable inorganic crystals are found the same way a chemist would find them by hand. You take a known crystal, keep its symmetry and the arrangement of its atomic sites, and swap in different elements. The hard part is choosing which elements go where. A prototype with four distinct sites and about eighty usable elements allows roughly forty million joint substitutions. Whether a swap survives depends on the other atoms in the crystal, not only on the element being replaced. Current tools either score elements one at a time with fixed similarity rules, or generate whole crystals from scratch with left-to-right language models. We propose a third option borrowed from text editing. A bidirectional masked model is trained on crystals written as short symbolic sequences: the space group, then for each site its symmetry label and its element. "Bidirectional masked" means the model is trained to fill in hidden tokens using everything around them, as BERT does for words. Its fill-in probabilities then serve as a context-aware proposal and plausibility prior for multi-site element substitution.

The core bet is narrow and testable. Single-site swaps are cheap to enumerate exhaustively, so the masked model has to earn its keep where enumeration breaks down. That happens in two places. The first is joint substitutions across several coupled sites. The second is out-of-distribution chemistries, where a learned stability predictor (the "surrogate" that stands in for expensive quantum calculations) is least trustworthy. There, a model that has learned which element combinations co-occur in which site environments should catch surrogate errors that an energy score alone misses.

Core Claim

The coordinate-free description of a crystal is its space group plus a list of Wyckoff positions, which are sets of symmetry-equivalent sites, each with an occupying element. In this representation the arrangement of sites behaves like a sentence, and choosing an element for a site behaves like lexical substitution: the right answer depends on the whole context. We claim three things. First, a masked model over Wyckoff sequences, queried with partial rather than full masking, proposes substitutions that are more often thermodynamically stable than context-free substitution statistics fitted to the same data. Second, joint masking of several sites finds stable multi-site substitutions that enumeration-plus-surrogate pipelines miss at equal compute. Third, combining the model's likelihood with a surrogate energy raises discovery precision over the surrogate alone, mainly on chemistries far from the training set.

Where This Sits

The mechanism comes from natural language processing. BERT-ATTACK and BAE showed that masking a target word and sampling from BERT's fill-in distribution yields replacements that fit their context far better than dictionary synonyms. We reuse that mask-and-infill loop directly. BERT-based Lexical Substitution identified a dilemma that maps cleanly onto chemistry. A full mask erases the original word and invites meaning drift, while no mask makes the model just predict the original. Its fix, randomly zeroing parts of the target's input embedding, reached 51.1% Precision@1 on SemEval 2007 against a prior best of about 40.8%. We adopt that fix, but in a structured form. The element embedding is built from physical descriptors, following the period/group/block/valence "subatomic tokenization" of Crystalite. We can then corrupt chosen descriptor groups, for example keeping valence-shell information while hiding period. This asks the model for "an element with the same chemistry but a different size." CLARE extended substitution with insert and merge operations, reaching 79.1% versus 63.4% attack success on AG News. We treat insertion of a Wyckoff site only as a secondary, explicitly riskier experiment. The detoxification work behind CondBERT warns that masked infilling tends to propose "opposites." We expect the chemical analogue, elements that fit the slot but flip the charge balance, and we handle it with a hard charge-neutrality filter rather than hoping the model avoids it.

On the materials side, three lines of work bound the proposal. Kusaba et al. already frame crystal structure prediction as substitution into database templates. They use metric learning on fixed composition descriptors, rank the true structure in the top five 60% of the time, and estimate that about 80% of known stable crystals have an isomorph in current databases. That estimate motivates us: template substitution covers most of chemistry. Their model, however, scores whole formulas against templates and does not condition a site's element on the other sites. Wren supplies our evaluation logic and our main surrogate. It predicts formation energy from the same coordinate-free Wyckoff representation, reaching 38% precision on the WBM set (53% with uncertainty filtering). It also concedes that it cannot find structures without known prototypes, a limitation our substitution approach shares and does not claim to fix. The generative line of WyckoffTransformer and CrysTune tokenizes Wyckoff representations exactly as we do, but generates left to right from scratch. CrysTune documents that reinforcement learning toward stability collapses novelty.

We do not claim to solve that collapse in general. Substitution on scaffolds gives compositional novelty by construction and zero prototype novelty, and we report both separately. LLEMA is the closest system in spirit. It uses a prompted autoregressive LLM to mutate formulas under rules such as same-group substitution, scores them with surrogates, and reports a ~33.6% hit rate on wide-bandgap semiconductors versus 6.5% for MatterGen. LLEMA's mutations are prompt-level heuristics over free-text formulas and CIF files. Ours are learned conditional distributions over symmetry-constrained site tokens. Finally, AlloyBERT shows that masked pretraining on materials text helps property regression, but it never uses infilling generatively and has no symmetry information.

Research Program

We would build a small BERT-style encoder over canonicalized Wyckoff sequences. The sequence contains a space-group token followed by one token pair per occupied orbit (Wyckoff letter, element). Positional encodings are dropped so the model is invariant to orbit order, as in WyckoffTransformer. It is trained with masked prediction on Materials Project structures before a chronological cutoff. At inference we take a stable scaffold, choose a subset $S$ of orbits to edit, and corrupt their element inputs in one of three ways: a full mask, random embedding dropout at rate $\rho$, or structured descriptor dropout. We then sample joint fillings by Gibbs-style resampling over $S$. Each candidate $c$ is ranked by

$$ \text{score}(c) = \hat{E}_{\text{hull}}(c) - \lambda \, \text{PLL}(c), $$

where $\hat{E}_{\text{hull}}(c)$ is Wren's predicted energy above the convex hull, $\text{PLL}(c)$ is the masked model's pseudo-log-likelihood (the sum over edited orbits of the log-probability of each element given all others), and $\lambda \ge 0$ is tuned on a validation split. Setting $\lambda = 0$ recovers surrogate-only ranking, which makes the prior's contribution directly measurable. Candidates that fail oxidation-state charge balance are removed before scoring. Survivors are converted to 3D structures with pyXtal and relaxed with a universal interatomic potential. Where budget allows, we follow up with DFT.

The analysis component asks what the model has learned. One question is whether its substitution probabilities for a given element pair change with site symmetry and co-occupants: does Li→Na become more or less probable depending on the anion? Another is whether structured dropout produces controllable shifts, such as size changes at fixed valence. If the conditional distribution does not vary with context, the central premise fails, and this diagnostic tells us early.

Baselines

The naive reference point is random element substitution on the same scaffolds under the same charge-balance filter, with a same-periodic-group swap variant that mirrors LLEMA's design rule. Both are scored by Wren and given the same candidate and relaxation budget. They show how much any learned prior adds over chemistry-textbook heuristics.

The ablation that isolates our key ingredient, context, is context-free pairwise substitution statistics. Here the probability that element $a$ replaces element $b$ is estimated from co-occurrence on shared prototypes in exactly the same training split, then combined with Wren using the same scoring formula and the same tuning of $\lambda$. This is the reviewer's strongest "simpler" competitor. If our masked model does not beat it, the bidirectional context buys nothing. The second ablation compares masking modes (full mask, unmasked, random dropout, structured dropout) and single-site versus joint editing.

The strongest existing scoring pipeline is Wren applied to exhaustive single-site enumeration, plus enumeration of multi-site combinations up to the same compute cap as our sampler. Wren's reported WBM precision is 38%, or 53% with uncertainty filtering (Wren). Our method must exceed the uncertainty-filtered Wren pipeline when we re-run it under identical splits and budget. Re-running matters because the published figure uses a different candidate pool.

Two other external baselines complete the comparison. For template-based substitution we compare against Kusaba et al.'s metric-learning method, which reports top-five structure recovery of 60% on its 35-case benchmark. For whole-crystal generation we compare against WyckoffTransformer samples filtered by Wren, given equal numbers of candidates; it reports 39.2% S.U.N. on MP-20. LLEMA's ~33.6% hit rate is on a task-specific benchmark that is not directly comparable. We will run LLEMA on our property-targeted variant if code is available, and otherwise report it only as context. Beating a baseline means a higher fraction of candidates confirmed stable among the top $k$ at equal relaxation budget, with bootstrap confidence intervals that do not overlap.

Why not just…?

Why not just enumerate every substitution and let Wren rank them? For single-site edits this is cheap and probably hard to beat, and we expect it to roughly tie our method there. The threat is real, so the decisive comparisons happen on two and three jointly edited orbits. There enumeration grows as about $80^{|S|}$ and must be truncated. They also happen on held-out chemistries, where surrogate errors grow. If the $\lambda > 0$ prior gives no gain over $\lambda = 0$ in either regime, we will report that the masked model is redundant with the surrogate.

Why not just use data-mined substitution probabilities? Pairwise statistics are context-free in the same way static synonym tables are, and Kusaba et al. motivate their method by the failure of pairwise substitution to account for the whole formula. That argument, though, is about formula-level context, not site-level context. Our context-free ablation, trained on identical data, settles the question directly. If it matches the masked model, we drop the claim about bidirectional context.

Why not just ask an autoregressive Wyckoff model or a prompted LLM to infill? An autoregressive model can infill by sampling and rescoring, but each joint fill then requires conditioning on sites that come later in the sequence, which it cannot do natively. CrysTune shows that steering such models toward stability collapses diversity. LLEMA's prompted mutations work well, yet they are rule-driven, and they emit free-text CIF files that need heavy filtering downstream. We include WyckoffTransformer-with-rescoring as a baseline at matched sample counts. If it matches our precision on joint edits, bidirectionality is not the active ingredient.

Proposed Experiments

The primary experiment is retrospective discovery on a chronological split. We train on Materials Project entries before a cutoff, in the style of MPTS-52, and apply every method to pre-cutoff stable scaffolds. We then measure the fraction of top-$k$ candidates that match post-cutoff stable entries, alongside precision against hull energies computed by the potential. The second experiment uses WBM, which was itself generated by element substitution on Materials Project prototypes and carries DFT labels. Each method ranks the WBM candidates that derive from each scaffold, and we report precision at $k$ against DFT stability. This tests the prior as a ranker using ground-truth labels, with no new calculations. The third experiment is prospective. A fixed budget of candidates per method, restricted to two- and three-orbit edits in under-explored chemical systems, is relaxed with the OMat24 EquiformerV2 potential, which reports F1 0.916 on Matbench Discovery (OMat24). The top candidates then go to DFT. We report compositional novelty and prototype novelty separately so the result cannot hide prototype reuse. A stretch experiment tests CLARE-style insertion of one new orbit and checks whether any inserted candidates are stable at all. We report it as a negative result if they are not.

The decisive checks are three. The first is whether context-conditioned probabilities beat the pairwise ablation on WBM precision. The second is whether the combined score beats Wren alone specifically on joint edits and on held-out chemistries. The third is whether structured descriptor dropout produces the intended controllable shifts in ionic size or valence. A clear failure on the first check ends the project's central claim. A failure on the second alone would leave a useful but narrower result: a better substitution proposer that adds nothing as a prior beyond the surrogate.